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NATURE AND PURPOSE OF STUDY 

This study was designed and written to help meet certain basic needs of 
various school personnel, and others, who conduct various types of survey 
research studies. It has evolved from an analysis of the^ actual survey tasks 
in which these people engage. 

Survey instruments of many kinds are being used extensively in most states 
for gathering needed data which may be relevant in the solution of various 
educational problems. Time, effort, and money are involved in all such studies. 
In some instances, the net result of an expensive and elaborate survey may be 
little more than another report to be added to the educator’s administrative 
library. To be assured that a study will have meaning and value and leave as 
little as possible to chance, the entire survey process necessitates painstaking 
planning of the overall study, careful designing of the survey instrument, 
insightful interpreting of the survey data, and accurate reporting of the survey 
findings. 

Purpose of Study 

A preliminary study of the problem of designing usable survey instruments 
revealed these persistent needs: 

1. The need for a better understanding of the preliminary considerations 
which are pertinent in planning a survey. This would include an under- 
standing of the function of the survey method in research as well as the 
values that may accrue when this method is used'. 

2. The need for classifying and defining the various types of survey instru- 
ments that may be used to collect needed or useful data. 

3. The need to make survey instruments more nearly valid and reliable 
through a better structuring of the instrument and the writing of the 

' itenis that make up the content of the instrument. 

4. The need for assessing the overall adequacy of the survey instrument 
before it is used. 

6. The need for improved reporting of the results of survey research. 

The purpose of this study is to meet these needs through the formulation, 
definition, and presentation of evaluative criteria and evaluative criteria test 
items which can be used to serve as guide lines in the design, the construction, 
and the evaluation of survey instruments and survey reports. Survey instru- 
ments which are carefully designed and constructed and which are evaluated 
before use in accordance with prescribed criteria should help to provide more 
nearly valid and reliable data. The end result will be a report that is relevant 
to the educational problem that initiated the investigation. 

Scope and Procedure of the Study 

The following steps or procedures were followed in preparing this report: 

1. A collection was made of various survey instruments which had been 
used in California as well as in other states in seeking answers to various educa- 
tional problems. Letters requesting samples of survey instruments which had 
been used in survey studies were sent to the following educational groups: 
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Nature and Purpose op Study 



Group 



Number 



California 

District Superintendents 
City Superintendents 
County Superintendents 



462 

90 

53 



Other States and Territories 

Chief Education Officer or Officers 

TOTAL 



56 



More than 300 survey instruments and completed survey reports of various kinds 
were collected by this means. 

2. The survey instruments and reports collected were studied and analyzed. 
An attempt was made during the analysis to isolate desirable elements which 
would form the nucleus for the development of evaluative criteria. An attempt 
was made, also, to classify the types of instruments which had been used in the 
various survey studies. This classification forms the basis for one of the chapters 
included in this report. 

3. An intensive study was made of the published literature since 1930 in an 
effort to isolate ard consolidate additional evaluative criteria. 

4. The next step was to classify, define, illustrate, and present the evaluative 
criteria in a form which would be of value to research investigators who use 
survey instruments for gathering needed data. These evaluative criteria, together 
with evaluative criterion test items for each of the criterion, were classified 
according to the needs outlined under the purpose of the study. 

5. The overall report was outlined in accordance with Step 4, and the five 
chapters which follow were prepared. 



DIRECTIONS FOR USING EVALUATIVE CRITERIA 

The criteria given are intended as guides for designing and improving 
survey instruments. (See Summary Highlights and Evaluative Criteria Sheets 
preceding each of the chapters of this report.) For each criterion, several 
criterion test items are given to aid the evaluator in making value judgments 
about the degree to which the particular instrument is fulfilling the criterion 
listed. A scoring system is provided for each test item. The directions for using 
each of these features follow: 

The Criterion 

The individual criterion should be rated by drawing a circle around the 
symbol that best describes the judgment of those making the evaluation of the 
survey instrument. The basis for this judgment should be the thinking 
results from a careful evaluation of the criterion in terms of each of the 
criterion test itema given with the criterion. The rating of the criterion should 
be done after the individual test items have been evaluated. 

Criterion Test Items 

Following the statement of each criterion in the chapter summary high- 
lights, some suggested criterion test items are listed to help the evaluator decide 
if the listed criterion is being met and to what degree. 

Provision is made after each criterion test item for the evaluator to rate 
that particular item according to the symbols listed below. 

After all the criterion test items have been examined and rated, a general 
evaluation of the criterion should be made. This should be based on the criterion 
test items, but it should represent an overall general evaluation rather than an 
average of the criterion test items since these test items may not be of equal 
weight in the evaluation of a specific survey instrument. 

Evaluative Criteria and Criterion Test Item Rating Scales 

The symbols following each criterion have these explanations for the evalu- 
tion or rating to be given to the criterion and the criterion test items : 

“0” Indicates that the criterion or the test item is not met and that 
additional work is needed to improve the particular item. 

“1” Indicates that the criterion or the test item is only partially met and 
that steps need to be taken for improvement. 

“2” Indicates that the criterion or the test item is met about half as well 
well as it could be if the instrument were considered as ideal. 

“3” Indicates that the criterion or the test item is adequately met but 
there is some room for improvement. 

“4” Indicates that the criterion or the test item is fully met. 

Note : If the criterion test item does not apply, it should be left blank and 
should not be included in the overall criterion evaluation. 



Example of Use 

Criterion — The principles of good grammar are not violated in writing the 
' items for the survey instrument. 






The rating of “2” is not an attempt to average the various criterion test 
items but rather it is the evaluator’s overall judgment regarding this 



vii 



viii 



Directions for Using Evaluative Criteria 



criterion. This rating is made after each of the criterion test items has 
been applied. It indicates that the instrument needs additional reworking 
if it is to be considered as grammatically correct. 

Criterion Test Items 

~ Rating 

1. Are the alternatives listed for multiple-choice items parallel 

in construction? ^ 

The rating of “1” indicates that the test item is only partially 
met and that steps need to be taken for improvement. 

2. Are all statements complete? 4 

The rating of “4” indicates that the test item is fully met. ■ 

3. Is there a consistency of grammatical person in the 

questions ? ^ 

The rating of “0” indicates that the test item is not met and 
that additional work is needed to improve the particular item. 



General 

The various criteria and the criterion test items are intended to serve these 
purposes : 

1. To provide guide lines for the researcher in each of the following areas: 

a. In planning the survey. 

b. In selecting the appropriate type of survey instrument to use. 

c. In writing the items comprising the survey instrument. 

d. In assessing the adequacy of the completed instrument before it is 
used. 

e. In reporting the results of the survey research. 

2. To guide the researcher and others in making a general overall evaluation 
of the survey instrument. 

3. To stimulate the desire to improve all aspects of the survey method of 
research so that adequate and meaningful data will be gathered. 

4. To save time, effort, and money in the conduct of the research study or 
investigation. 

Evaluative Criteria Summary Sheet 

After each of the criterion listed for a particular aspect of survey instru- 
ment has been evaluated, the evaluative criteria summary sheet on page 86 
should be completed. This is not to be an averaging of the criterion test items 
for the particular criterion, but rather, it is the evaluator’s overall judgment 
of how well the criterion has been met by the survey instrument. This evaluation 
sheet will be partially completed before the survey instrument is used; an addi- 
tional part of the evaluation sheet will be completed as the data are being 
collected as a check on the adequacy of the data collected; and finally, the 
remainder of the summary evaluation sheet will be completed after the research 
has been completed and the report written. The completed summary evaluation 
sheet will then serve as an overall measure or rating of the particular research 
study and it may indicate improvements that need to be made in future studies 
of a similar nature. 

In addition to the foregoing analysis, the research report evaluation form 
given in Chapter V, page 82, should be completed before the research report 
is written in its final form. This form may be used, also, to evaluate a completed 
report to indicate areas of weaknesses and/or strengths of the report. 
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CHAPTER I— PREREQUISITES TO PLANNING A SURVEY 

NTRODUCTION 

I. THE SURVEY DEFINED 

Definition— The collection of accurate and meaningful data con- 
cerning a contemporary topic. 

II. THE PRIMARY VALUES OF THE SURVEY 

To provide a basis for making comparisons and determining trends 
To reveal current weaknesses and/or strengths of a present situation 
To provide information for making decisions 

[II. LIMITATIONS OF THE SURVEY METHOD 

Reveals, at best, “what is” rather than “what ought to be” 

Does not reveal the causal factors influencing behavior or attitude 
Cannot be used for securing all types of evidence which may be 
needed for wise decision making 

Susceptible to limitations in the areas of sampling and adequacy of 

evidence gathered . , 

Possibility of securing inaccurate information or of making incorrect 

interpretations of data collected 
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IV. PRELIMINARY CONSIDERATIONS IN PLANNING A SURVEY 6 

Criterion— The survey is so planned, organized, coordinated,^ and 
^ controlled that accurate and meaningful data which are 
relevant to the question or problem raised are collected. 

0 12 3 4 



Criterion Test Items 

1. Has the problem to be studied been identified and 
clearly stated? 

2. Is there a clear statement of the purpose of the 
survey ? 

3. Has a thorough and thoughtful study been made of 
* related materials and literature? 

4. Is the survey method most appropriate for the prob- 
lem to be investigated? 

5. Are the data desired unavailable elsewhere?, 

6. Have the sources for the desired data been carefully 
identified? 



Rating 
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Pages 

7. Has the scope of the undertaking been carefully 

delimited? 

8. Are competent personnel and adequate financial re- 
sources available for the study? 

V. SOME ILLUSTRATIONS ^ 

A Community Study of Business and Distributive Occupations 
A Follow-Up Study of a Work Experience Program 
A Survey of a New Position in the School System 



VI. SUMMARY 
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CHAPTER I 



PREREQUISITES TO PLANNING A SURVEY 

Finding answers to problems has become an occupational responsibility for 
many wbrkers in the field of education. As the task of the school has becoine 
more comprehensive, the need to seek answers is intensified. No longer is it 
adequate to look at the traditional sources of information — the tomes that 
contain records of the past. Evidence upon which decisions can be made and 
problems solved must be secured from contemporary sourc^. 

The guidance department of a school is interested in securing parental 
opinion on the subject of homework. The guidance director makes a list of 
questions that he wants answered, develops an appropriate data-gathering 
instrument, and carries through his survey. 

A school has developed a new work experience program. After an experi- 
mental period of three years, the school desires to determine the effectiveness 
of this new aspect of the program. The task is to secure answers to such 
questions as: “How valuable has the experience been from the students’ point 
of view?” “How satisfied are the businessmen with the workers they have been 
receiving?” “What should the school now do — continue the program, modify 
the program, or delete it?” A group of teachers who have been working with 
the new program are given the responsibility for making a survey. 

Students participated in an Outdoor Education in Science and Conservation 
program. The program was experimental in nature. The directors and their 
instructors wanted an evaluation of the program by the students enrolled. They 
chose a committee to develop a plan for a survey that would include responses 
from all participants as well as detailed reactions from a limited sample of the 
participants. 

An array of situations could be enumerated to illustrate the occasions when 
surveys are appropriate for securing needed information. To be efficient in the 
gathering of needed data, a survey must be carefully planned and carried 
through. Much time and attention is needed to transform the vague, often 
general questions that initiate the desire for additional information into a 
refined plan with carefully written instruments, sufficiently sensitive to elicit 
the desired data. 

I. THE SURVEY DEFINED 

Although there are many variations of the survey, a simple definition might 
be: the collection of data concerning a contemporary topic. A survey may be a 
very simple undertaking as the determining of the number of children under 
six years of age in a school district. On the other hand, a survey may be as 
complex as probing into the social attitudes of a sample and meeting the rigorous 
requirements of any research undertaking. The data may be secured through 
mailed questionnaires or through face-to-face interviews. 

Good and Scates, in discussing one type of survey, the descriptive-survey- 
status research, provide a type of definition: 

. . . this method is essentially a technique of quantitative description of the 
general characteristics of the group. This approach to problemsolving seeks 
to answer questions as to the real facts relating to existing conditions. , . . 

Many surveystatus studies emphasize present conditions with an impli- 
cation of the idea that things will change. This vital interest in trends as 
the dynamics of status is in keeping with the general dynamic emphasis or 
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outlook of present thought on research methodology. Although^ the status- 
survey study may he made as a matter of ascertaining facts, it is usually 
(and normally) made against a background of interests, purposes, and 
established values, so that the facts at once are seized upon by these mental 
backgrounds and employed in larger schemes of thinking or in application 
to specific problem situations.^ 

II. THE PRIMARY VALUES OF THE SURVEY 

Knowledge of the current situation is valuable for a number^ of purposes. 
First, such knowledge can provide a basis of comparison if similar data are 
available from an earlier time. Trends or shifts can be noted and may provide 
clues for future action. The results of a survey undertaken to find out why 
professional staff members of the YWCA resigned their jobs, for example, was 
compared with a similar study done in 1926. The results of the 1958 study 
showed almost identical factors as did the earlier study. The committee, without 
the data collected, would not have believed that the same factors were still 
uncorrected in the present situation.^ 

Second, current weaknesses and strengths can be revealed^ through the 
knowledge of what the situation is presently. A study of the helping teacher in 
elementary school districts in the State of California was undertaken to deter- 
mine, among other things, the nature of the responsibilities of this emerging 
position and the strengths and weaknesses of the position.^ 

Third, survey data are often used as the basis for making decisions. For 
example, with the continuing agricultural revolution has come the need for a 
different type of training for future agricultural workers in the State of Cali- 
fornia. As a basis for determining what the requirements were, a survey was 
conducted in 14 selected farm-center communities in 1956. From this initial 
survey and reports from other states has come the decision to undertake pilot 
studies that will ultimately develop into permanent agricultural business cur- 
ricula in junior colleges.* * 

Good and Scates in discussing the value of the survey made the following 
comment: 

A descriptive-survey study may be thought of in its relation to continuous 
(regular, periodical) reports in any country or industry. A survey is simply 
one of these reports, a cross-section at a given time, but it is usually of 
particular importance or p'icsents special information; it may be the begin- 
ning of such reports or, if they are going as a series, may be extensive in 
character. Acting as an initiating agency the survey has a unique oppor- 
tunity to do two things: (1) to establish the value of facts or to show their 
value, so that regular reports may then later be called for; (2) to center 
attention on the most important things to be reported; it is a great challenge 
to ascertain the central elements or the dominant and fundamental factors 
that will indicate the general conditions.^ 



1 Carter V. Good and Douglas E. Scates, 
Appleton-Century-Crofts, Inc., 1964), p. 661. 

2 Marie Updegraff, “Why Do They Quit?” 



Methods of Research (New York: 
The YWCA Magazine (June, 1969), 



32. 

* John William Wright, An Analysis of the Emerging Position of Helping Teacher 
(No. 13 of a Series Approved by the Committee on Research Studies, Los Angeles City 
School Districts, Los Angeles, California), May, 1968. 

* S. S. Sutherland and 0. E. Thompson, Training Required by Workers in Agp^ 
cultural Business <md Industry (Progress Report, Pilot Studies to Develop Junior 
College Curricula in Agricultural Business, California State Department of Education) , 
1967 . 

* Good and Scates, op. oit., p. 662 . 
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III. LIMITATIONS OF THE SURVEY METHOD 

The survey method, by its very nature, does not provide directly the type 
of information about what is best. A picture of the most commonly 
patterns in the contemporary situation is not necessarily a picture of what is 
best in the situation. There are, of course, instances where decisions are^^^^^ 
Based on the concensus of the population surveyed. In much of the educational 
work, however, there is implicit in the research activity an endeavor to discover 
what is best or most appropriate. Data about the current situation are indeed 
helpful for clues as to possibilities for the future. The relaiionsh^ip, ^o^eve , 
between “what is” and “what ought to be” is not a simple, direct one. F ndings 
from surveys should not be used, therefore, to establish the best curriculum, the 

best program of activities, or the best practices. ^ 

A survey cannot probe into the causal factors influencing behavior or 
attitude. Some survey instruments are, of course, far more sensitive than others 
and do make an initial attempt at probing. As Good and Scates commented. 

The descriptive-survey or normative approach has not characteristiccdly 
delved deeply into interrelationships or causal factors. The typical survey 
has stopped with the disclosure of facts and a suggestion of relatively prom- 
inent possible connections between these facts and apparent causes, although 
more recently both educational and social surveys have shown an increased 
interest in the causes of the phenomena described.^ 

There are limitations concerning the nature of evidence that can be secured. 
A survey can r^eal how youngsters feel about the amount of hom^ork they 
are getting in business mathematics; a survey will not reveal the effeaiveness 
of the homework in increasing the learning of business mathematics. Through 
a survey, the attitude of students toward what kind of spellers they are can 
be determined; however, a survey cannot reveal how much skill the students 

possess in spelling difficult words. ^ i.-mi+o 

The survey method, like other methods of research, is susceptible to limita- 
tions in the areas of sampling and adequacy of evidence gathered. The preva- 
lence of such weaknesses demands that a few comments be made at this 

An efficiently designed survey requires that a sample no larger than that 
needed to*4epresent the population under study should be seeded. In ™any 
educational surveys there is an attempt to solicit responses^ from e o a 
population. Whether this is a wise way to undertake a survey is an appropriate 
concern. When the significant variables are known, an attempt should be made 
to employ sampling techniques.^ 

Another point in the process of completing a survey where attention must 
be given to the adequacy of the sample is at the time of the analysis of the 
returns. The investigator is plagued by the question: “Are those who respond 
to the questionnaire representative of the total group to whom questionnaires 
were sent?” Unless the investigator has evidence that the answer to the for- 
going question- is yes, he will not be able to make generalizations concerning the 
total group. An investigator cannot assume that his 20 or 30 or 40 per cent 
return represents his initial population. There are far too many instances when 
the assumption cannot be maintained. For example, in follow-up studies of 
graduates it is very probable that those who can report success in their after- 
graduation experiences will be most likely to return questionnaires. This same 
group may well evaluate their school experiences through rose-colored glasses. 



7 See^ Mildred' Parten, Surveys, Polls, and Samples: Practical Procedures, Chap- 
ters VII, VIII. and IX, especially. 
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A few of the very disgruntled may be moved to respond. However, the many 
who don^t return questionnaires should be encouraged to do so, or allowances 
need to be made for the silent ones. The caution noted in the review of a survey 
of the 1949 graduates of the Harvard Business School is . one that should be 
common to many surveys: 

Afore notes of doubt and bitterness would probably be heard if every 
member of the class had filled out the questionnaire. About 200 failed to do 
so, and (so alumni secretaries would speculate) this group probably includes 
most of the less successful members of the class. 

Among the missing may well be some who think Harvard was a great 
waste of time and who want nothing more to do with it.^ 

Another weakness is the possibility of receiving inaccurate information for 
a variety of reasons. The respondent may be motivated to “look good” or to 
appear respectable. Furthermore, it is very possible that a person does not know 
even the facts about his own experiences. Payne made the statement: 

The researcher can go astray by assuming, that people are more aware 
of the commonplace than they actually are. The phenomenon of unobserv- 
ance confronts us on every side. Its result is that answers come not in terms 
of facts as they exist but in terms of what the respondent thinks the facts 
ought to be. When we rely on the faulty memory of a cross section of people, 
the popular calendar designs, brands of beer, and breakfast foods appear 
even more popular than they are.* 

Jahoda, Deutsch, and Cook also comment on this topic: 

Hot only may there be lack of incentive to report openly one s beliefs, 
ings, motivations, plans, and so on, but in addition there may be an inability 
to make such reports. As psychoanalysts have pointed out, many of our 
important beliefs and motivations are inaccessible or unconscious: not being 
aware of them, we cannot report them.^* 

There must be tests made to determine the adequacy of the answers 
received. Such precautions will result in a more accurate interpretation of the 
data. 



IV. PRELIMINARY CONSIDERATIONS IN PLANNING A SURVEY 

Jahoda, Deutsch, and Cook succinctly identify the first step in a survey 
when they stated: 

... the first step in a survey, as in any other research, is to define the 
question which the survey is to answer. Unless the objectives are specified 
sufficiently to ensure that the data collected are relevant to the question 
raised, the survey may not provide the desired information. . . 



A further confirmation of the need for giving attention to certain factors 
is contained in Parten’s statement: 

*“The $14,000-a-Year Man,” Fortune (June, 1959), 116. 

•Stanley Payne, The Art of Asking Questioyis (Princeton: Princeton University 

Marie Jahoda, Morton Deutsch, and Stuart W. Cook, Research Methods in Social 
Relations (Part I: Basic Processes; New York: The Dryden Press, 1961), p. 164. 
Ibid., p. 60. 
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Designing, organizing, and conducting a survey if like establishing and 
running a business enterprise. Both require technical mowledge and skill, 
administrative ability^ and specific experience or training in work similar 
to that being organized. . . . Only by carefully planning the survey from 
start to finish can reliance be placed upon the results and in many cases, wiU 
the findings ever reach the publication stage.^^ 

Consideration 1: Has the problem to be studied been carefully stated? 
It is exceedingly important that the problem be clearly identified and stated so 
that there is no ambiguity about what the core of concern is to be. A statement 
of the problem together with a statement of the purpose will lay the groundwork 
for the further development of the plan of procedure. A problem carefully 
envisioned and identified should be stated in a hundred words or less in such a 
way that its dimensions are clear to the competent reviewer. 

Consideration 2: Has a thorough and thoughtful study been made of 
related materials and literature? Few surveys will represent exploration into a 
totally untapped field of knowledge or inquiry. The investigator charged with 
undertaking a survey should attempt to profit from the experiences of others. 
A careful and systematic search should be made for related references. A group 
of teachers who are planning to do a follow-up study of their high school , 
graduates will profit from reading similar studies that have been completed. 
Through a careful analysis of what has been done, the teachers will not overlook 
areas of relevance to the investigation under consideration.^ A review of ^e 
instruments used in securing data will be helpful to devising a more useful 
instrument for the investigation contemplated. 

The investigator must also study the related theoretical material available 
in libraries. A survey to be significant must reflect the very best evidence and 
theory that are available to date. To begin a survey on the motivations for 
college possessed by high school freshmen before reading and studying the more 
relevant books and monographs on the subject is to fail to achieve the potential 
values of such a survey. 

Every undertaking in the educational field ought to, in some respects, extend 
our basic knowledge or provide empirical evidence to confirm theories currently 

tenable. 

Consideration 3; Is the survey method most appropriate for the problem 
to be investigated? The survey method is not necessarily the one that is 
appropriate for the problem under investigation. In fact, a survey may add little 
enlightenment to the problem under study: For example, if a school is interested 
in finding out how the family influences the occupational aspirations of students, 
the investigator might feel that a survey among a sample of high school students 
would provide the data desired. However, a more careful study of the infor- 
mation sought might lead the investigator to a realization that a more valuable 
method might be an intensive interview study where it would be possible to 
probe into the somewhat complex cluster of factors that influence young people 
in the interest they show in an occupational field. 

A school may be interested in knowing if one method of teaching basic 
principles of accounting is more effective than another method. A survey to find 
out what method the schools in the state use and to determine the final results 
on an objective examination would indeed not be the effective way to find an 
answer to the problem. An experiment, carefully designed and executed, would 
be the method appropriate to the problem. 



12 Parten, op. cit., p. 48. 
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Consideration 4: Are the data desired unavailable? It is important to 
make a careful search of sources for the data that are desired. Many a high 
school principal interested in knowing what courses are offered in other schools 
or the number of students taking particular courses has devised a questionnaire 
and mailed copies to a sample of the schools. He could have received the infor- 
mation in many instances from his state department of education where " 
ment data are assembled, thus saving himself as well as other school officials 

time and effort. - . , 

Few people know, for example, that the Census Bureau will compile a variety 

of statistics at the cost of the actual job only. The Census Bureau has a wealth 
of data that can be provided in summary form. (Information on individuals 
is confidential and, therefore, never divulged.) 



Consideration 5: Have the sources for the desired data been carefully 
identified? Identification of the persons from whom data are to be gathered 
is important for the basis of decisions concerning the vocabulary level of the 
instruments to be used, the technique for eliciting data, and for other purposes. 
Also, the investigator must be sure that the sources identified are the ones from 
which he can get the most direct information. An investigator, for example, 
would not go to the teachers to find out what the enrollment of the school was 
— ^he would go to the principal who weuld generally have such data readily 

available. 



Consideration 6: Has the scope of the undertaking been carefully 

delimited? The investigator must establish boundaries around his survey. It is, 
unfortunately, generally not practical to undertake an omnibus study.^ If the 
investigator is interested, for example, in studying the work experience of 
students he must establish the boundaries of his survey. Is “work experience” 
to apply only to that work activity that is under the auspices of the school? 
Or, is the investigator interested in knowing the gamut of work experience that 
high school students have had? 

An investigator who wishes to find out the value of a camping experience 
for high school sophomores must decide whether the youngsters only are to be 
involved or whether their parents, teachers, and possibly others in the com- 
munity are also to be included in the inquiry. 

How much of the universe will have to be included to secure the kind of 
information sought? It is expensive, as well as unnecessary, to include every 
person in the universe in the survey. There are instances, of^ course, when there 
is an attempt to account for every person, but generally some number less than 
the total is sufficient to use as the sample. If a survey is to be done of talented 
youth in the secondary schools of California, for example, it will not be efficient 
or economical to send questionnaires to all youth who qualify for the classifica- 
tion “talented youth.” The investigator must determine how many will be in- 
cluded. He may want to consider the different types of schools and decide to 
choose all talented youth in each of a certain number of schools. The schools 
selected would be those that would be representative of all the schools in the 
state. 

If knowing how 10 per cent of a group feels gives a clear pattern of the 
feelings of the 100 per cent, there is no need to use the 100 per cent. Of course, 
the responsibility for determining adequacy of sample rests with the investigator. 
The bibliography includes several books on statistical methodology that will pro- 
vide assistance on the subject of sample selectiop. 



Consideration 7: Are adequate personnel and financial resources available? 
The undertaking of a survey requires skills of competent people in the area of 
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survey methodology. The competency required can be learned through inde- 
pendent study as well as through formal instruction or on-the-task experience. 
A well-envisioned survey will not develop at the hands of the uninitiated or 
unsophisticated investigator. 

Surveys cannot be done without financial resources. Inadequate funds may 
result in curtailment of the potential value anticipated in an undertaking. At 
the same time, it is possible to make wise use of limited funds when the original 
planning is done with intelligent awareness of the budget allocations. 

V. SOME ILLUSTRATIONS 

A Community Study of Business and Distributive Occupations.^® The 
Santa Monica Unified School District undertook an investigation of selected 
business and industrial establishments in the city for the purpose of determining: 

a. The number of part-time job opportunities in the business and distributive 
field for high school and college students. 

b. The number of full-time job opportunities in business and distributive 
occupations available to high school and college students at graduation. 

c. The type of training which the employers oi the community require for 
these various jobs. 

The report identified the ma’n questions of the undertaking as: 

a. What are the full-time business and distributive job opportunities for 
high school graduates and for two-year college graduates? 

b. What are the part-time business and distributive job opportunities for 
high school students and for students in a two-year college program? 

c. To what extent will the community support a cooperative training pro- 
gram at both the high school and college levels in the business and dis- 

‘ tributive occupations ? 

d. To what extent will the community support a general work experience 
program at the high school level in the business and distributive 
occupations? 

e. What are the basic training needs in the field of business and distributive 
occupations ? 

f. What is the general attitude of the employers concerning business and 
distributive education training? 

The fnvestigation was limited to the job opportunities and training needs 
of the following types of enterprises: 

a. Retail trades (omitting all liquor stores, second-hand stores, and non-store 
retailing) 

b. Wholesale establishments 

c. Service establishments (omitting bars) 

d. Professional oflSces 

e. Industrial plants 

The data were collected through personal interviews with officials in each 
of the five types of enterprises included in the investigation. 



Taken from a Survey of Business and Distributive Occupations, Santa Monica 
Unified School District, Santa Monica, California (June, 1957). 
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A Follow-Up Study of a Work Experience Program.. To what extent hrv3 
the work experience been a valuable phase of the student’s education? Has the 
work experience program affected drop-out rates, vocational planning, social 
maturity, scholastic attainment, attendance records, citizenship records of 
students? Did the graduates of the program possess sufficient preparation in 
basic and special skills required for the jobs they were assigned to do? Is the 
work experience program one that should be continued ? The foregoing questions 
initiated a survey by the Torrance Unified School District to determine the 
effectiveness of their work experience program. 

The oflScials felt that although there were large numbers of students partici- 
pating, the success or failure of the program must be assessed in other ways. 
The investigators prepared two questionnaires. One was to be used in eliciting 
reactions from students; the other was to be used in seeking opinions of 
employers.^* 

A Survey of a New Position in the School System. There was little avail- 
able information concerning the relatively new position, that of helping teacher, 
in the elementary school districts in the state of California. There was a general 
feeling that more effective use would be made of personnel assigned to such a 
position if the duties, status, and related factors of the position were known. 

The investigating personnel outlined a series of broad questions that would 
form the basis for the questionnaire. The questions were: 

a. What school districts have establsihed the position of helping teacher? 

b. What is the basis for selection and assignment of personnel to the 
position? 

c. What specific duties and responsibilities are assigned to personnel in the 
position ? 

d. What factors have contributed to the establishment of the position? 

e. What incentives exist or are provided to encourage teachers to apply for 
the position? 

f. What modifications should be made in further development of the position? 

Following a review of the related literature and interviews in three school 
districts that employed helping teachers, the committee developed a tentative 
questionnaire. A pilot study was made to determine the adequacy of the instru- 
ment developed. Districts where help'ing teachers were employed were identified 
through a preliminary questionnaire. Teachers in the position as well as admin- 
istrators responsible for supervising the position were participants in the 
survey.^* 

The foregoing are merely three illustrations of the nature of surveys done 
in the educational setting. In each instance a lack of information that was 
believed to be valuable was the motivation for the investigation. 

VI. SUMMARY 

The survey is a useful method for securing certain types of data that a 
schoo’ may need to make wise decisions. Prerequisite to a successful survey is 
the need to answer adequately the questions: 



^♦From_A Follow-Up Survey Report of Distributive and Diversified Work Experi- 
ence Education Programs, Publication No. 39, Torrance Unified School District, Tor- 
rance, California (January, 1968). 

^®John William Wright, An Analysis of the Emerging Position of Helping 
Teacher (No. 13 of a Series Approved by the Committee on Research Studies, Los 
Angeles City School Districts, Los Angeles. California, May, 1968). 
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a. Has the problem to be studied been carefully stated? 

b. Has a thorough and thoughtful study been made of related materials 
and literature? 

c. Is the survey method most appropriate for the problem to be investi- 
gated? 

d. Are the data desired unavailable? 

e. Have the sources for the desired data been carefully identified? 

f. Has the scope of the undertaking been carefully delimited? 

g. Are adequate personnel and financial resources available? 



SUMMARY fflGHLIGHTS 
AND 

EVALUATIVE CRITERIA 

CHAPTER II— TYPES OF SURVEY INSTRUMENTS 

Pages 

INTRODUCTION 14 

I. EXPLORATORY SURVEY INSTRUMENT 14 

Criterion — The exploratory survey instrument is designed to get 
initial or tentative answers to the problem. 

0 12 3 4 

Criterion Test Items 

Rating 

1. Are most of the items open-ended so as to allow for 

freedom of response? 

2. Will the items elicit the gamut of responses that are 

appropriate to the topic under investigation? 

3. Will the items be intrinsically interesting to the 

prospective respondent? 

4. Is the instrument designed, primarily, to secure 

qualitative data? 

6. Is the instrument to be sent to persons who will 

give insightful answers? 

Examples (Pages 15-16) 

II. FACT-FINDING SURVEY INSTRUMENT 16 

Criterion — The fact-finding survey instrument is designed to secure 
facts about a particular situation. 

0 12 3 4 

Criterion Test Items 

Rating 

1. Are the questions so worded as to secure objective 

information? 

2. Since the possible answers are already known to the 

investigator, have the items been so arranged as to 
make it easy for the respondent to check the appro- 
priate blank or blanks? 

3. Do the questions secure objective information about 

a prevailing situation ? 

4. Do the questions ask for information in the manner 

in which it is generally maintained? — 

5. Are the questions stated precisely and will they 

mean the same thing to all respondents? 

Examples and Illustrations (Pages 18-20) 

12 



Evaluative Criteria 



13 



Pages 

III. OPINION-ATTITUDE SURVEY INSTRUMENT 21 

Criterion — ^The opinion-attitude survey instrument is designed to 
secure subjective data that provide an accurate descrip- 
tion of current feelings and opinions. 

0 12 3 4 

Criterion Test Items 

Rating 

1. Has the instrument been developed on the assump- 

tion that the respondents have an opinion or attitude 
toward the topic under study? 

2. Are the items sufficiently sensitive so as to elicit an 

accurate opinion or attitude from the respondent? 

3. Will the respondent feel that he has complete free- 

dom to answer truthfully without fear of repercus- 
sions to what he records as his opinions? 

4. Are the items organized and arranged in a psycho- 
logically appealing order? 

Examples and Illustrations (Pages 23-25) 



IV. THE EVALUATIVE SURVEY INSTRUMENT 



Criterion — The evaluative survey instrument, while similar to the 
opinion attitude survey instrument, attempts to assess the 
worth, the value, or the success of some phenomenon; 
hence, it should elicit value judgments from the respond- 
ents. 

0 12 3 4 

Criterion Test Items 

Rating 

1. Have the pertinent factors concerning the phenom- 
enon to be evaluated been identified and isolated? 

2. Is the number of points at which the respondent may 
■ evaluate no greater than the degree of differentiation 

possible in the situation? 

3. Is the number of differentiating points as few as is 
necessary to provide an adequate evaluation in the 

given situation? 

4. Are the items constructed and organized so as to 

preclude the possibility of the “halo effect” in 
responses ? 

Examples and Illustrations (Page 27 and pages 29-32) 
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CHAPTER n 



TYPES OF SURVEY INSTRUMENTS 

Any attempt to classify survey instruments ends somewhat short of being 
satisfactory. Mutually exclusive and comprehensive) exhaustive categories have 
yet to be devised. Overlapping appears inevitable. Instruments have been classi- 
fied according to content such as school-census taking, administrative practices, 
classroom teaching techniques, and so forth. A few moments of reflection will 
cause the reader to realize that such a categorization is cumbersome because of 
the wide variety of content possible for the survey method. Another classifica- 
tion system sometimes used is that based on the group for which the instrument 
is intended; for example, the instruments for administrators, for teachers, for 
students, for parents, and for other groups. This classification also seems some- 
what inadequate since- the variable “group for whom intended” does not appear 
to be as critical as other variables when instruments are under study. 

For purposes of the discussion in this chapter, survey instruments are 
divided into four broad categories based on the general purpose for which the 
instrument is to be developed. The four categories are: the exploratory instru- 
ment, the fact-finding instrument, the opinion-attitude instrument, and the 
evaluative instrument. This categorization, too, does not fulfill the requirements 
of a totally satisfactory division. However, it appears adequate for the discussion. 

There will be occasions when the instrument used is a combination of two 
or more t3npes. When a survey attempts to discover what the current practices 
are in a particular education situation as well as opinions of what are the best 
practices, the instrument would be a combination of the fact-finding and opinion- 
attitude survey instruments. The investigator would, in such a case, give con- 
sideration to the criteria for the two types of instruments. 

I. EXPLORATORY SURVEY INSTRUMENT 

The exploratory survey, as the name implies, attempts to discover initial 
or tentative answers to the problem under discussion. Such an undertaking 
generally represents the first venture into the area under study. The evidence 
available about the topic is generally exceedingly limited. The possible answers 
are not known to the investigator. Therefore, the instrument used will have 
little structure. 

CRITERIA FOR OVERALL PLANNING 

The items are usually open-ended. Because of the paucity of evidence or 
information about the topic under discussion, the questions must be wide open 
so that respondents will have the freedom to answer in the fashion that seems 
appropriate to them. 

The items must elicit the gamut of responses that are appropriate to the 
topic under investigation. The investigator must keep uppermost in his mind the 
question: “Will the question as worded cause the respondent to include all the 
aspects of the topic that are relevant?” A question such as, “How valuable do 
you feel your high school experiences were?” is not a good question if the 
investigator wants students to be sure to include their reactions to non-classroom 
activities as well as classroom activities. “High school experiences” will mean 
only the classroom experiences to many students. 

The items must be intrinsically interesting to the prospective respondents. 
Open-ended questions demand considerable thought and attention on the part 
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cf the respondent. People tend to dislike writing out answers to questions. 
Only those who are highly motivated to do so will take the time and effort to 
answer free-response items. It is, therefore, very important that the investigator 
choose respondents who possess the requisite interest. 

The exploratory instrument is most useful for securing qualitative data. 
The nature of answers to free-response, or open-ended questions, makes quantifi- 
cation difficult, if at all possible. The value of this type of instrument in those 
instances where quantification is desirable is highly questionable. The explora- 
tory instrument is often a phase of a pilot study that leads to an instrument with 
more structure and, thereby, possible quantification. The responses to the ques- 
tion, “Why did you decided to send your child to the Outdoor Science Camp?” 
may in a final instrument become a multiple-choice item: 

“Check the reason or reasons for sending your child to the Outdoor Science 
Camp; 

He has shown. interest in science 

He had never attended a camp and we felt he should have such 

an experience 

etc.” 

The reasons listed as alternatives were secured through an analysis of the. free 
responses in the pilot or exploratory study. 

Insightful persons should provide answers to the items. This criterion is 
somewhat related to the characteristic of need for interest on the part of 
respondents; however, there is a different focus here. If an investigator, for 
example, is interested in determining the behavior patterns of talented students 
in the classroom situation through an exploratory survey with the teachers, he 
will want to choose those teachers who are sensitive to such patterns in their 
classrooms. There would be many teachers who would not have observed such 
differences among the students. 

Jahoda, Deutsch, and Cook made a comment about the experience survey, 
which is in many ways an exploratory survey, that is relevant to the foregoing 
discussion; 

Research economy dictates that the respondents in an experience survey 
he carefully selected. The aim of the survey is to obtain insight into the 
relationships between variables rather than (as in some other types of 
surveys) to get an accurate picture of current practices or a simple con- 
sensus to the best practices. Thus the respondents must be chosen in terms 
of their likelihood of offering insightful contributions. In other words, a 
selected — not a representative — sample of people working in the area is 
called for. 

In an experience survey it is a waste of time and effort to interview 
people who have little competence, or little relevant experience, or who lack 
ability to communicate their experience.^ 

EXAMPLES 

A study entitled “The Bases on Which Teachers Are Recommended for 
Dismissal” used an exploratory instrument. It was the attempt of the investi- 
gator to analyze the experience of selected school administrators in relation to 



Marie Jahoda, Morton Deutsch, and Stuart W. Cook, Resewrch Methods in Social 
Relations (Part I: Basic Processes; New York: The Dryden Press, 1961), p. 87. 
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the universal problem of teacher retention and dismissal. Henderson’s questions, 
which were listed one to a page, follow: 

1. What steps are followed to reach a decision of dismissal in your school? 

2. Please list the items most consistently brought to your attention as reason 



for dismissal. , , . . , 

3. In your relationship with teachers that were recommended for dismissal, 

please relate one experience that verified that the techniques used in dismissal 

were sound and successful. , , . • i 

4. In your relationship v/ith teachers that were recommended for dismissal, 
please relate one experience in which the teacher involved challenged your 

procedur^.^^^^ of dismissal that have gone to the board or the court, what 
procedures have your school or your district established to be certain that the 

case would be well presented? ^ i. i. 4 . 

6. In cases of dismissal that have gone to the board or to the court, what 

points in procedure have been challenged by the teacher, the board, or the 
court 

7. In cases of dismissal that have gone to the board or to the court, what 

reasons have consistently stood up as being valid? ^ 

8. Please list all cases that have gone before the board for a hearing or to 

court for trial.* 



What is being done for the gifted child in the school district was the basis 
for an exploratory study in the San Gabriel School District. The teachers to 
whom the questionnaires were sent were given a list of characteristics of gifted 
children. The list gave the teachers a kind of operating definition for gifted 
child.” The introductory statement and the items on the second page of the 
instrument follow: 

1. We realize that a formal program has not been instituted in this district, 
but we are eager to know of any classroom procedures that you have found 
successful in working with children who meet the criteria outlined on page 1. 
Please feel free to make comments in any or all of the following areas: 



a. Mathematics 

b. Social Studies (Including science and all related subjects) 

c. Aesthetic Development (Art, Music, Language Arts) 

d. Personality Development (Development of ethical sensitivity Moral and 

Spiritual Values) * 



A review of the items included in the foregoing illustrative survey instru- 
ments will reveal the need for insightful, interested respondents. At the same 
time, it is clear that the questions allow considerable flexibility to the respondents 
and should give the investigator valuable qualitative data that may or may not 
be conducive to statistical analysis. 



II. THE FACT-FINDING SURVEY INSTRUMENT 

There is frequently the need to secure the “facts” about a situation. A 
superintendent of schools may want to know whether elementary school^ teachers 
or secondary school teachers have the longer teaching day j a principal may 
want to know to what extent the core curriculum is found in other junior high 



2 From unpublished questionnaire forms and covering memorandum by Wayne L. 
Henderson, Mt. Diablo Unified School District, Concord, California. u • i 

•From a questionnaire prepared by the .San Gabriel School District, ban Gabriel, 

California (November, 1957). 
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schools in his state; the board of education may want to know how many children 
of school age and under school age live in the community. Indeed, the types of 
“facts” sought of school personnel would make a long list. 

criteria for overall planning 

The information sought is objective. Would another person who had access 
to the source of evidence provide the same answers to the questions asked in a 
fact-finding instrument as did the actual respondent? The courses tirnght in the 
core curriculum in the junior high school, the number of children in a particular 
household, the salaries paid beginning teachers — these and similar data are 
readily ascertainable, providing, of course, that such vital figures are maintained 
by the groups in question. In contrast to such data can be cited data that are 
not objective in nature; for example, the values of the core program to the total 
school’s curriculum as determined by the teachers, opinions concerning what the 
beginning teacher’s salary should be, or the estimates of average family size 
made by a sample of the community’s residents. 

The possible answers to the questions are generally already known to the 
investigator. Unlike the exploratory survey instrument, wherein the investigator 
is probing for answers, the fact-finding survey instrument is aimed at finding 
out how many responses will be iriade on each item. The possible answers are 
known. When the question: “JIow many children are there in this household 
who are under school age ?” is asked, it is possible to list the alternatives 
beginning with 0 and running to 5 or 6. The question, “Are you using high 
school students for part-time positions in your company?” will be answered with 
a yes or no. 

Since answers are anticipated, questions may be dichotomous, as in the case 
of the yes or no question, or multiple choice, as in the case of the first example 
given in the preceding paragraph. 

The primary purpose of the fact-finding instrument is to determine the 
prevailing situation. Census-taking is the aim of the fact-finding instrument. 
What is the range of salaries found among school maintenance personnel? How 
many different subjects do secondary school social studies teachers teach during 
a year? What are the non-class responsibilities of teachers? These are some of 
the general questions that will be basic to the formulation of the fact-finding 
instrument. 

The questions should ask for information in the manner in which it is 
generally maintained. The investigator must have considerable knowledge of 
the manner in which data sought are maintained by the respondents so that 
answers to questions can be easily recorded. If figures must be derived, there is 
less tendency to cooperate. If the salary of the beginning clerical worker is 
wanted, the investigation should indicate the choices according to weekly wages 
if it is the practice in the particular community to pay such workers weekly. 

The need to state questions precisely is of primary importance. To secure 
data that are comparable the investigator must be certain that the items in 
the fact-finding survey are interpreted in the same way by all the respondents. 
Items must be stated precisely. An investigator interested in knowing how 
many workers are in a number of different job classifications needs to specify 
whether he wants actual numbers or percentages. Respondents may, unless given 
instructions, respond with either numbers or percentages. Such varied returns 
would not be comparable. Many beginning workers when asked if they partici- 
pated in an orientation program when they entered the companies in which they 
are employed said “no.” However, further exploration revealed that they had had 
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an orientation program — it had been called an induction program, an introduction 
to your job, etc. Words that may have special meanings need to be illustrated 
or defined so that the complete meaning becomes clear to the respondent. 

EXAMPLES 

A fact-finding instrument may be one question only. Such a brief instrument 

is given in Illustration 1, page 18. j 

Shutt, in an attempt to determine the leave of absence policies for certificated 
personnel in California School Districts, submitted a fact-finding instrument to 
the school districts. Illustration 2, page 19, gives a portion of the questionnaire 

he used. ... 

Schwartz devised a fact-finding instrument to discover the organization 

and administration of the junior high school instructional program. A part of 
his 9-page instrument is shown in Illustration 3, page 20. 



ILLUSTRATION 1 



TAX RATES ^ 

Please insert your basic school tax rates for current operations, 1958-59. 
(Exclude special taxes for community services, retirement, bond redemp- 
tion, etc.) 



Legal Rate 

Maximum Levied 



Elementary district basic tax rate 

High school district basic tax rate 

Junior college district basic tax rate 

Information supplied by: 

Name 

Position 



1 Adapted from fact-finding survey of Oakland Public Schools, Oakland, California. 
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ILLUSTRATION 2 



LEAVE OF ABSENCE POLICIES FOR CERTIFICATED PERSONNEL 
IN CALIFORNIA SCHOOL DISTRICTS ^ 



Section II. Special Types of Leaves 
IMPROVEMENT LEAVES 
Sabbatical — - General Information 

6. Who is eligible for sabbatical leave (provided the seven year requirement 
has been met) ? (a) — Teachers, (b) Administrators, (c) Not granted. 

6. Who makes the routine recommendations for approval of sabbatical leave 

applications? (a) — Director of personnel, (b) Supt. (c) Asst. Supt. 

(d) — Leave committee, (e) Other: 

7. Number in order (Number 1 being highest) the basis for priority of 

applicants in granting sabbatical leaves, (a) Length of service, (b) 

Time since last leave, (c) Purpose of leave, (d) Date of application. 

(e) Other: 

8. What percent of the total number of teachers are permitted on sabbatical 

leave at one time? (a) — 1 %. (b) — 2%. (c) 3%. (d) Other: 

9. Have applications ever exceeded the allow number? (a) ^Yes. 

(b)_No. 

10. Is a priority list established after leave requests are in to indicate an order 
for granting requests? (a) ^Yes. (b) No. 



15. What pay is received by administrators and supervisors on sabbatical leave? 

(a) — None, (b) — Pay minus replacement, (c) One-half regular pay. 

(d) — Full pay. (e) Other: 

16. What limit is placed on additional income earned by teachers on sabbatical 

leave? (a) — No additional employment permitted, (b) Up to equivalent 

pay of district, (c) — No limit, (d) Other: : 

17. What written policies cover situations when sabbatical plans are interrupted 

by unforeseen events ? (a) None, (b) Leave postponed, (c) Cancelled. 

(d) Other: 



^From questionnaire form U§ed by Noel M. Shutt, principal, Los Angeles City 
School. 
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ILLUSTRATION 3 



A Research Study on 

The Organization and Administration of the 
Junior High School Instructional Program ^ 



II. ORGANIZATION OF INSTRUCTION 



Exploratory Courses in Practical Arts and Fine Arts 

Courses offered to permit pupils to discover their interests and capacities to help 
guide them in their future educational program and vocational activities. 



1. Exploratory Courses 



Circle Less than 10-15 16-20 One 

Grades 10 weeks weeks weeks Year 



Shop classes — boys 7-8-9 

Shop classes — coeducational 7-8-9 

Homemaking classes — girls 7-8-9 



Homemaking classes — 
coeducational 

Music classes 

Art classes 



7-8-9 

7-8-9 

7-8-9 

7-8-9 

7-8-9 



Have no exploratory courses in practical 

2. Primary objective of shop courses 3. 
for boys 

( )a. Pre-vocational training 

( )b. Vocational training 

( )c. Exploratory experience 

( )d. General education 

( )e. Other (specify) 



arts or fine arts ( — ). 

Primary objective of homemaking 
courses for girls 

( )a. Pre-vocational training 

( )b. Vocational training 

( ) c. Exploratory experience 

( ) d. Training for parenthood and 

family life 

( ) e. General education 

( ) f. Other (specify) 



5. Features of your orientation program for new pupils 

( )a. Orientation presented in 

block-time classes 

( )b. Orientation presented in 

English classes 

1 Page 3 from the instrument used by Dave Schwartz, principal, Pacoima 
High School, Pacoima, California, for a Research Study on the Organization and Ad- 
ministration of the Junior High School Instructional Program. 
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Page 1 of the Distributive Education Survey form devised by the San Jose 
Unified School District and the Bureau of Business Education, California State 
Department of Education, provides the purpose of the undertaking, plus a series 
of fact-finding questions. A part of the first page of the 2-page form is repro- 
duced in Illustration 4, below. 

III. THE OPINION-ATTITUDE SURVEY INSTRUMENT 

Numerous occasions arise in the American school system when there is need 
to determine how individuals feel on some issue. What do students think of the 
extra-class activities provided for them? What do parents think of the home- 
work assignments their youngsters receive? How does the beginning teacher 
feel about the resources that are available to him? These are merely illustrative 
of the many questions that arise daily that initiate the opinion-attitude type of 

"Pereas the fact-finding instrument seeks objective, easily -verifiable data, 
the opinion-attitude survey instrument seeks subjective data that provide an 
accurate description of current feelings and opinions. 

CRITERIA FOR OVERALL PLANNING 

The instrument must be developed on the assumption that the respondents 
have in some fashion developed an opinion or attitude toward the topic under 
study. This assumption must be ever evident in the planning for an opinion- 
attitude survey. The validity of the responses depends on the fulfillment of this 
requirement. Teachers in Los Angeles would not be asked to express an opinion 
on a core program for the schools of southwestern Kentucky. 



ILLUSTRATION 4 

Business Men: Section No. 

Distributive Education Survey ^ 

This survey is being made by the San Jose Unified School District and the Bureau 
of Business Education, California State Department of Education. 

The purpose of this survey is to determine the extent to which a Distributive 
Education Work Experience program would be valuable to the business men of 
San Jose and to the students of the San Jose high schools. 

In a Distributive Education Work Experience program the student takes a 
business course related directly to work in a store. The content of the course 
frequently includes such topics as working relationships, personality and sales, 
good working habits, good-will and its meaning, and the handling of confidential 
business affairs. While taking the course the student works in some retailing 
or service establishment for an average of 15 or more hours per week (includes 
Saturday work) . The student is checked periodically to ascertain if the best 
principles of salesmanship are being applied by the student to the job. This 
checking is done by a representative of the school who is especially traineu by 
education and actual work experience for this type of coordination. 

Students are given school credit toward graduation and are paid an hourly wage 
by -the employer. 

The cooperation of your establishment in helping us with this survey will be of 
benefit to the school district. And we hope through the results of the survey we 
may find ways to help you. 

^ From A DistTihutive Occupations Survey, San J ose, California. Conducted and 
reported by Ross W. Atkinson, Co-Sponsors, San Jose Unified S^ool District and 
Bur6EU of Businoss EJducfition, Cslifornis i^tEtG DGpsrtmGnt of Educfition, opring 
Semester, 1957. 



22 



Evaluative criteria 



1 . 



Name of Business: 



2. Person interviewed and position: 

3. Address: High School Area: 



4. Number of full-time sales employees: M W. 

5. Number of part-time sales employees: M W. 



9. Number of full-time clerical employees: M W 

10. Number of part-time clerical employees: M W 

Which age groups are you willing to hire? 16-19, 19-22, 23-40, 41-up. 



The items must be sensitively developed to elicit the most accurate opinion 
or attitude of the respondent. The questions must not be so stated that one 
answer more than another would tend to be selected. Such biases will indeed 
distort the opinions or attitudes of the respondents. . , j 4.1, 

Another consideration at this point is the need to be sure to include the 
range of possible opinion that might be expressed by the respondents. This 
would include making provision for the expression of “no opinion,” unless, o_ 
course, there is the desire to force the respondent to express a specific opinion. 

Furthermore, there must not be in the letter accompanying an opinion- 
attitude schedule any statements that cause the respondent to feel that one par- 
ticular set of opinions would be favored. 



Respondents must have complete freedom to answer truthfully. To be valid, 
the responses must be made with complete freedom. The respondent^ must 
believe that there will be no repercussions to what he records as his opinions. 
An opinion survey in which the respondents feel coerced is indeed a farce. 
If the principal of a junior high school, for example, wishes to know how the 
teachers feel about the new core program, he must establish the fact unequiv- 
ocally that he desires their real feelings. If he has all along the way been a 
staunch supporter of the program, he may find it somewhat more difficult than 
it would otherwise be to establish the proper rapport to secure real feelings on 
the subject under study. 

The items must be organized in a psychologically appealing order. The 
task of the investigator of an opinion or attitude survey includes the ordering 
of the items in a manner that will aid the respondent in thinking about the 
subject of the instrument. Parten, in discussing question arrangement, made 
the comment: 

For example, in one study of the attitudes of people toward advertising, it 
was possible to get the respondents to think in terms of advertising toward 
which they approved without actually mentioning advertising. They were 
asked about dresses. Then when the question on advertising was asked they 
thought of dress advertising and responded more favorably to the idea of 
advertising than when the dress question was omitted.^ The question or ques- 
tions preceding a given question may produce a certain "set'* in the inform- 
ant and cause him to reply^ very differently from the way he might if the 
preceding questions wore different.* 

* Mildred Parten, Surveys, Polls, and Samples: Practical Procedures (New York: 
Harpers, 1960), p. 213. 
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EXAMPLES 

Illustration 5, below, and Illustration 6, page 24, are excerpts of opinion- 
attitude survey instruments. Illustration 5 is a part of page 1 of an instrument 
used to determine how well the high school met some of the needs and interests 
of students. A Parent Attitude Inventory is shown, in part, in Illustration 6. 
The purpose of the survey as given in the instrument was to obtain an indica- 
tion of parent’s attitudes toward children. 

Illustration 7, page 25, is a portion of Poll No. 2 used by the Metropolitan 
School Study Council of New York City to determine what the public felt the 
schools could do. 



ILLUSTRATION 5 



My Opinions ^ 

1. As you think back over the past school year, how would you rate the help you 
received from your teachers in each of the areas listed below ? 

Very Little or 

Helpful Helpful No Help 

a. Selecting my high school subjects 

b. Planning my life work (vocational guidance) 

c. Selecting activities that interested me (ath- 
letics, clubs, dramatics, etc.) 

d. Preparing to meet requirements for getting 

into college 

e. Learning how to go about getting a job 

f. Helping me learn how to solve problems of 

a personal kind (worries, home problems, 
getting along with others, etc.) 

2. As you think back over the past school year, how would you rate the help you 
received from your counselors in each of the areas listed below? 

Very Little or 

Helpful Helpful No Help 

a. Selecting my high school subjects 

b. Planning my life work (vocational guid- 
ance) 



f. Helping me discover my abilities, interests, 
and weaknesses 

g. Helping me learn how to solve problems of a 
personal kind (worries, home problems, get- 
ting along with others, etc.) 



1 From form used by Long Beach Unified School District, Office of the Director of 
Research, June, 1968. 
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ILLUSTRATION 6 



PARENT ATTITUDE INVENTORY ^ 

Directions 

This inventory consists of 120 statements designed to sample opinions about 
parent-child relations. There is considerable disagreement as to what these 
relations should be; therefore, there are no right or wrong answers. What is 
wanted is your individual feeling about the statements. Read each statement 
and decide how YOU feel about it. Then mark your answer on the space provided 
on the answer sheet. Do not make any marks on this boOiClet. 

SA A U D SD 

■ I I I I I I I I 

SA A U D SD 

I I ■ I I I I I I 

SA A U D SD 

I I I I ■ I I I I 

SA A U D Sb 

II II I I ■ II 

SA A U D SD 

II I I I I I I ■ 

D — Disagree 
SD — Strongly disagree 



If you strongly agree, blacken space under “SA” 

If you agree, blacken space under “A” 

If you are undecided or uncertain, blacken space 
under “U” 

If you disagree, blacken space under “D” 

If you strongly disagree, blacken space under “SD” 

SA — Strongly agree U — Undecided or 

A — Agree uncertain 



1. Most children are obedient. 

2. Children who “act smart” probably have too high an opinion of themselves. 

3. Minor disciplinary situations should sometimes be turned into jokes. 

4. Shyness in children is preferable to boldness. 

6. Most children don’t appreciate what a parent does for them. 

6. A child’s companionships can be too carefully supervised. 

7. A child should keep his likes and dislikes to himself. 

8. It sometimes does a child good to be criticized in the presence of other 
children. 



13. There is too great an emphasis upon children “keeping quiet.” 

14. A child’s failure is seldom the fault of the parents. 

15. There are times when a parent cannot be blamed for losing patience with 
a child. 

16. A parent should never discuss sex problems with the children. 



1 From a Pa/rent Attitude Survey, El Segundo Unified School District, El Segundo, 
California, March 19, 1957. 
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ILLUSTKATION 7 



A STUDY OF PUBLIC OPINION ABOUT SCHOOLS^ 



Poll No. 2 

What Do You Think Schools Could Do? 

We want to find out what you think public schools could do. We do not necessarily 
mean the schools which you attended or the schools which your youngsters 
attend. We are interested in schools as they could be if they were 
best practical methods of instruction that you know about. What could schools 

do over a period of time about the problems listed below? 

Read each statement and put a circle around the letter or letters which com- 
plete (s) the sentence. 

N— Nothing AL— A Little M— Much VM— Very Much 

1. Schools could do N AL M VM to improve the physical health of youngsters. 

2. Schools could do N AL M VM to improve the mental health of youngsters. 

3. Schools could do N AL M VM to prepare youngsters who, as adults, will 

be able to manage their own family business 

affairs. 

4. Schools could do N AL M VM to show youngsters how to spend leisure 

time happily. 

6. Schools could do N AL M VM to aid students to select the vocation for 

which they are best suited. 

6. Schools could do N AL M VM to produce an American people who can 

“see through” propaganda and misleading 
information. 

7. Schools could do N AL M VM to produce an American people who can vote 

critically and intelligently. 

8. Schools could do N AL M VM to improve interfaith and race relations. 



16. Schools could do N AL M VM to give the American people an understand- 
ing of and respect for law. 

16. Schools could do N AL M VM to give youngsters an understanding of 

what is required of them as young people 
and later as adults in happy home and family 
living. 



1 From form A Study of Public Opinion about Schools, prepared by Metropolitan 
School Study Council, New York, New York. 
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IV. THE EVALUATIVE SURVEY INSTRUMENT 

The judgment of participants or observers is often needed to make wise 
decisions about further action. Educational bodies are interested in answers to 
questions such as: “To what extent has the workshop fulfilled the expectations 
of the participants?” “How well does the supervisor feel the new teacher adjusts 
to the requirements of his job?” “How valuable do graduates feel high 

school education was in preparing them for the jobs they are now in . 

An evaluative survey, on careful analysis, is in many respects a type of 
opinion-attitude survey. However, there is one significant difference that seems 
to justify separate treatment of the evaluative survey. This difference is: an 
evaluative survey attempts to assess the worth, the value, or the success of 
phenomenon. In other words, a value judgment, an appraisal, is the core of the 
responsibility of the respondent to an evaluative survey instrument. 

CRITERIA FOR OVERALL PLANNING 

The pertinent factors concerning the phenomenon to be evaluated must 
be identified. Possibly, the most critical consideration of the investigator in an 
evaluative survey is to isolate the factors that will provide a valid appraisal. 
In any situation, there will be a multitude of factors that in some manner influ- 
ence the activity. To choose the relevant factors demands a lucid image of the 
dominant influences. If an investigator wanted to devise an instrument -for the 
evaluation of the beginning teachers in the school district, it is his first task to 
determine what aspects of the new experience should be questioned. 

The number of points at which the respondent may evaluate should be no 
greater than the degree of differentiation possible in the situation. If the 
respondents have had occasion to determine only whether or not the person 
being evaluated accomplished the task, the items might have no more than two 
alternatives. One alternative would indicate that the individual did complete 
the task; the other that he did not. Now, if the situation were different and 
the observer was able to determine how well the task was completed, the investi- 
gator will have justification for asking the quality of the work done on a three-, 
four-, or possibly, five-point scale. 

The number of differentiating points should be as few as is necessary 
to provide an adequate evaluation. As the number of points identified on the 
continuum increases so increases the difficulty of making judgments. ^ It is much 
simpler to say that Student A is a satisfactory student and Student B is not satis- 
factory than it is to say that Student A should be given a rating of 7 on a 
10-point scale, while Student B should be given a rating of 3 on the same scale. 
To determine if Student A belongs really at 6 or 7 or 8, or even 9, is far more 
difficult than to merely say he is satisfactory. 

A corollary to the difficulty of determining the most accurate evaluation is 
the difficulty of achieving a high degree of reliability in evaluations. To use the 
same illustration, if a group of 10 teachers were asked to evaluate 20 students as 
satisfactory or unsatisfactory, there is considerable assurance that most of the 
20 students would be evaluated in the same way. However, if we increase the 
points from two (satisfactory— not satisfactory) to ten, our responses would 
show considerable variance. The assumption here is that the evaluations by each 
of the ten teachers would be made independently of the others. Somewhere 
between the two, which may not provide sufficient differentiation, and the ten, 
which may be far too many for reliable evaluations, the investigator inust choose 
a number that strikes a satisfactory balance between precision of evaluation 
and reliability of the evaluation. Three, four, and five gradations are most com- 
monly used for evaluative instruments. 
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The items should be written so as to preclude the possibility of the “halo 
effect” in the responses. There has been considerable evidence to support the 
observation that a person in evaluating another individual is often influenced 
by his general feeling toward the individual and tends to evaluate the person in 
the same manner on a variety of factors. As one technique to discourage such £ 
generali 2 ed evaluation, the investigator can list the points in different orders, as 
illustrated in the following two questions: 

How much initiative has the student shown in the work he has done in your 
office? 

no opportunity to observe 

none 

a limited amount 

a reasonable amount 

a great deal 

How much responsibility has the student assumed in doing the work in 
your office? 

a great deal 

a reasonable amount 

a limited amount 

J none 

% 

no opportunity to observe 



EXAMPLES 

In an attempt to discover how parents of Palo Alto rated the schools, an 
evaluative instrument was mailed to a random sample of the parent population. 
Page 1 of this 2-page form is given in Illustration 8, page 29. 

Page 1 of a 3-page evaluation form used to determine the effectiveness of 
the beginning teacher in the Petaluma City Schools is shown in Illustration 9, 
page 30, 

In Illustration 10, page 31, are included the first and fourth pages of 
the evaluative instrument used to determine criteria for evaluation of student 
newspapers in public junior colleges. 

V. SUMMARY 

An overall consideration of the criteria appropriate to the survey instru- 
ment to be developed is exceedingly important. The details of construction must 
be undertaken within the framework of the general criteria if the final instru- 
ment is to be effective. The criteria identified in this chapter are restated here: 

The Exploratory Survey Instrument 

1. The items are usually open-ended. 

2. The items must elicit the gamut of responses that are appropriate to 
the topic under investigation. 

3. The items must be intrinsically interesting to the prospective respond- 
ents. 

4. The exploratory instrument is most useful for securing qualitative 
data. 

5. Insightful persons should provide answers to the items. 
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The Fact-Finding Survey Instrument 

1. The information sought is objective. 

2. The possible answers to the questions are generally already known to 
the investigator. 

3. The primary purpose of the fact-finding instrument is to deter- 
mine the prevailing situation. 

4. The questions should ask for information in the manner in which it 
is generally maintained. 

5. The need to state questions precisely is of primary importance. 

The Opinion-Attitude Survey Instrument 

1. The instrument must be developed on the assumption that the respond- 
ents have in some fashion developed an opinion or attitude toward 
the topic under study. 

2. The items must be sensitively developed to elicit an accurate opinion 
or attitude of the respondent. 

3. Respondents must have complete freedom to answer truthfully. 

4. The items must be organized in a psychologically appealing order. 

The Evaluative Survey Instrument 

1. The pertinent factors concerning the phenomenon to be evaluated 
must be identified. 

2. The number of points at which the respondent may evaluate should 
be no greater than the degree of differentiation possible in the 
situation. 

3. The number of differentiating points should be as few as is necessary 
to provide an adequate evaluation. 

4. The items should be written so as to preclude the possibility of the 
“halo effect” in the responses. 
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ILLUSTRATION 8 



PARENT EVALUATION OF PALO ALTO SCHOOLS ^ 

Directions : Please rate the Palo Alto School District as YOU see it by placing 
an “X” in the appropriate square after each of the items listed below. 



We do not expect that all parents will make their decisions on the basis of 
complete information, but we do want your opinion of our schools based on what- 
ever information you presently have. 





SUPERIOR 


GOOD 


AVERAGE 


BELOW 

AVERAGE 


POOR 


better 
than most 
schools 


better 
than many 
schools 


about 
the same 
as most 
schools 


not as 
good as 
many 
schools 


not as 
good as 
most 
schools 


1. Teaching reading 












2. Teaching arithmetic 












3. Teaching creative 
writing 












4. Teaching writing 
skills, such as spell- 
ing, punctuation, 
grammar 












6. Teaching social 
studies, such as his- 
tory, geography, eco- 
nomics 












6. Teaching science, 
natural and physical 












7. Teaching good study 
habits and study 
skills 












8. Teaching library and 
reference techniques 












9. Developing in stu- 
dents a favorable 
attitude toward 
thinking 












To. Encouraging critical 
thinking 












11. Choosing good 
teachers 












12. Providing adequate 
materials in the way 
of books, art sup- 
plies, libraries, 
audio-visual aids 












13. Offering special 
services, such as psy- 
chologist, audiologist, 
speech therapist, 
dentist 













1 From form Parent Evaluation of Palo Alto School*, prepared by the Palo Alto 
Unified School District, Palo Alto, California. 
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ILLUSTRATION 9 

PETALUMA CITY SCHOOLS 
Petaluma, California 

TEACHER EVALUATION 1 

Name 

School 

Evaluation Year 

Date 

Evaluation by Principal 

ZZZI Joint Evaluation by Teacher and Principal 

Suggestions for Use 

This instrument is: 

1. to be presented to new teachers immediately 

2. to be accompanied by a conference with the principal 

3. to be used by all probationary teachers twice each school year and by 
permanent teachers each three years, or more often if requested by them 

4. designed by teachers for the improvement and growth of the individual 

Personal characteristics— A professional teacher has a wholesome personality, 

a sound character, enjoys good physical and emotional 

health, and shows that he: 

displays real enthusiasm in teaching 

is self-assured with humility 

is adaptable and can accept new ideas 

has a good sense of humor 

is objective about himself 

is punctual 

is free from annoying mannerisms 

exercises initiative 

is consistent 

uses good English 

has good physical health 

has high ethical and moral standards 

is well groomed and dresses appropriately 

is friendly, enthusiastic, tactful 

has dignity, poise, and self-control 

Teacher-Staff Relationships — A professional teacher is a good team worker who 

is conscious that his attitudes affect all others on 
the school staff. He is loyal to the school's program 
and policiv“s and gives evidence that he: 

is a good team worker , „ i 

shows responsibility toward the whole school as well as to own class 

is friendly to the whole staff 

is able to disagree pleasantly 

does not talk unfavorably about other school personnel, parents, or stu- 

de nts in an unprofessional way 

1 From form Teacher Evauation, prepared by the Petaluma City Schools, Petaluma, 
California. 



KEY 

1. Very good 

2. Satisfactory 

3. Improvement needed 
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ILLUSTRATION 10 



CRITERIA FOR EVALUATION OF STUDENT NEWSPAPER 
in Relation to Educational Objectives 
of the Public Junior College ^ 

Inquiry Form 

Data Regarding Evaluator . Please provide the following information: 

Name Date 

Position 



Address 

Introduction. On the following pages are criteria for evaluating a student 
newspaper in terms of major educational objectives of the public junior college. 
It is intended that these criteria later be developed into a check list which may 
assist newspaper staffs, advisers, deans of student personnel, and others in deter- 
mining whether such publications contribute to the over-all functions of the 
institutions they represent. 

As a leader in either junior college education or in the field of journalism, 
you are asked to help decide which “evaluative criteria” are most important, 
which should be eliminated, and whether any additional criteria should be 
included. 

Instructions . Please take the following steps in the examination of this 
material : 

a. Read the criteria and circle one of the numbers to the right of each item 
in ^ accordance with its approximate value to the particular junior college 
objective. The key for this purpose is as follows: 

5 — Of particular importance to the objective 
4 — Of “above average” significance 
3 — Of “fair” value 
2 — Of some use to objective 

1 — Of no importance to junior college objective under consideration; 
should be eliminated from criteria. 

b. In the event that you wish to suggest additional criteria, please list them 
at the end of the section to which they are related and give each item a weighting 
of^ from 2. to 6. in the same manner that you have already marked the other 
criteria. Use additional sheets of paper if necessary. 

c. Make any general observations, suggestions, or criticisms on the last page. 

d. Return the criteria list and comments to Johns H. Harrington, Los 
Angeles City College, 855 N. Vermont Ave., Los Angeles 29, in the stamped, 
self-addressed return envelope which is enclosed. 

Summary of results . Thank you very much for your valuable guidance! A 
summary of the results of this study will be sent to you if you wish. However, 
it may be six months or more before the study is completed. 

1 From form Criteria for Evaluation of Student Newspaper, developed by Johns H. 
Harrington, Los Angeles City College, Los Angeles, California. 
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II. VOCATIONAL EDUCATION 
(Terminal Objective) 

The student newspaper contributes to the achievement of the junior college 
objective of vocational education by publishing: 

Circle one (values range from 
1, “of no importance,” to 
5, “of particular importance) : 



1. Descriptions of vocational curricula 

2. Descriptions of opportunities in vocational 
fields 

8. Reports of current employment requirements 
in vocational fields 

4. Announcement of assistance to high school 
students and/or parents in making plans for 
completing vocational curricula 

6. Announcement of counseling or other serv- 
ices provided by the junior college to assist 
incoming students in planning programs of 
training for vocational fields 

6. Descriptions of activities of campus organ- 
izations concerned with students majoring 
in two-year terminal curricula, such as : 

a. Air Transportation 

b. Business 

c. Dental Assistants 

d. Radio Broadcasting 

e. Secretarial Science 

Other (List coverage of any other groups 
which contribute to vocational objec- 
tive.) : 

f 

g. 

h. 

7. Reports of visits by representatives of local 
business or industry to interview or assist 
prospective employees 

8. Announcement of work-study arrangements, 
employee benefits, or other special advantages 
offered by certain vocational fields 

9. Announcement of performance or recognition 
of junior college graduates as a group in cer- 
tain businesses or industries 

10. Reports of activities of alumni of the junior 
college who are employed by business and 
Industry 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 

1 2 3 4 5 
1 2 3 4 5 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 
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SUMMARY fflGHLIGHTS 
AND 

EVALUATIVE CRITERIA 



CHAPTER III— WRITING ITEMS FOR SURVEY INSTRUMENTS 

Pages 

INTRODUCTION ’ 

I. TYPES OF ITEMS ^ 

Open-Ended Questions 

Criterion— The questions are carefully developed so as to elicit 
meaningful answers which express beliefs, opinions, 
or knowledge of the respondents. 

0 12 3 4 



Criterioil Test Items 

1. Has the meaning of the question to the respond- 
ent been considered? 

2. Has the superiority of the open-ended question 
for seeking the desired data been established 
and tested? 

8. Does the question ask for one item of infor- 
mation only? 

4. Has the scope of the question been indicated? 



Rating 



Multiple-Choice Items 

Criterion— The questions are clearly stated so as to elicit the 
choice of the most appropriate alternative listed with 
the question. 



Criterion Test Items 

1. Do the choices or alternatives listed provide 
the variety of possible responses to each item? 

2. Is the number of responses allowed the respond- 
ent clearly indicated? 

3. Are the alternatives provided mutually exclu- 
sive? 

4. Has the number of choices been limited to no 
more than six or seven items? 



Rating 



II. GENERAL CONSIDERATIONS IN WRITING ITEMS 
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Language 

Criterion— The words used convey the intent of a particular 
question and will mean the same thing to all respond- 



ents. 



0 12 8 4 
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Criterion Test Items 



1. Have the words used for conveying the intent 
of a particular question been carefully chosen 
and tested? 

2. Has the general vocabulary level and the famil- 
iarity of the respondents with the topic under 
investigation been kept in mind as the words 
used were selected? 



Rating 



Grammar 

Criterion — The principles of good grammar are not violated 
in writing the items for the survey instrument. 

0 12 3 4 



Criterion Test Items 



1. Are the alternatives listed for multiple-choice 
items parellel in construction? 

2. Are all statements complete? 

3. Is there a consistency of grammatical person 
in the questions? 



Rating 



Physical Arrangement of Items 

Criterion — The physical arrangement of the items will be 
attractive to the respondent and will expedite the 
tabulation of the data when the survey instruments 
are completed and returned. 

0 12 3 4 



Criterion Test Items 



1. Has sufficient space been allowed for write-in 
answers ? 

2. Are the items so arranged that the answers 
appear in as straight a column as possible? 

3. Has a variety of type styles been used for 
emphasis and for ease in reading items? 

4. Is the finished instrument neat and orderly? 

6. Does the finished instrument convey the impres- 
sion that it is relatively easy to complete? 



Rating 



er|c 



CHAPTER in 



WRITING ITEMS FOR SURVEY INSTRUMENTS 

The art of writing items. that are effective is not, unfortunately, one that 
is easily acquired. Unlike Humpty-Dumpty, who could make words mean any- 
thing he wanted them to mean, the investigator must set as his goal the use of 
words that will mean the same thing to all who read them. Of course, such a gom 
is actually achieved in few instances since the varied backgrounds with which 
individuals read words inevitably influence their interpretations. However, 
attention and effort directed toward writing clear, succinct items will add 
immeasurably to the success of a survey. 

I. TYPES OF ITEMS 

OPEN-ENDED QUESTIONS 

The open-ended questions, also identified as the free-response or free-answer 
questions, are simply questions which are used most frequently to elicit answers 
that are not in any way influenced by possible alternatives. There are occasions 
when the investigator is interested in discovering what people believe on their 
own. In such cases this type of question is especially useful. For example, if an 
investigator wants to determine the personal attributes personnel directors feel 
are most important in the young people they hire, the investigator may choose to 
use an open-ended question. He may be more concerned with the attributes named 
when no list is provided than he is in tallying the number of times each attribute 
on a given list is checked. 

Because open-ended questions require considerable writing on the part of 
the respondent, they tend not to bring as many returns as do questions where 
checking or underlining only is required. The open-ended question should be 
avoided if the investigator wishes to get comparable data. Respondents may fail 
to include aspects of their feelings or knowledge because they feel these aspects 
are obvious or so common that the investigator would not be interested in them. 

If the questions are to be used for interviews, the open-ended type is often 
very satisfactory. The somewhat ambiguous open-ended question is helpful in 
probing beliefs, opinions, or knowledge possessed by the interviewee. Since 
people tend to enjoy talking far more than writing, the extent of response is 
no objection in the interview situation. 

The meaning of the question to the respondent must be considered. 
Although there are occasions when the investigator is curious about the re- 
sponses of individuals to an ambiguous question, generally the investigator has 
in mind a particular type of information. It is important that the investi- 
gator answer adequately the question ! “What will these words mean to the 
respondent ?” 

Example : “How far is- your company from downtown?” 

Comment: What did the investigator want? Did he want to know the 
’ mileage? the time it would take to travel by bus? by car? on 
foot? Would the respondent know where “downtown” began? or 
ended? Did the investigator mean by “downtown” the corner of 
Main Street and Cross Town Highway where the city buses 
load and unload passengers for the mid-town shopping area? 
Or did he mean anywhere in the central shopping area, the 
boundaries of which seem to be growing less exact as the town 
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rapidly expands? A possible item to secure the information 
desired is: 

“How much time is required to drive from your company to 
the Mid-Town Parking Center? 

less than 30 minutes 

from 30 minutes to less than an hour 

an hour or more” 

Example : “What plans do you have for what you want to do when 'you 

graduate from high school?” 



Comment : This is the first question of a section headed “Vocational Am- 
bitions” on a counseling survey questionnaire. What does the 

investigator reallj/ want to know? How va luablft is. it to. know 

“plans” without finding out “What you want to do.” Can the 
investigator assume that the respondent in telling about plans 
will either deliberately or inadvertently reveal what it is he 
wants to do? If the latter is important, should it not be elicited 
in a more direct manner? Two items might more effectively 
secure the information desired. Suggested items to replace the 
one given in the example are: 

“What would you like to do vocationally when you graduate 
from high school?” 



“What plans have you made to pursue your vocational desires?” 



Example ; ^*What was your first full-time job after high school graduation?” 



Comment : Is the investigator wanting to know the first permanent full- 
time job when the respondent began his working career or the 
first full-time job immediately accepted upon high school gradu- 
ation? Will the graduate who took' a temporary full-time job 
for the three months of summer record that job or will he list 
the job he took on a permanent basis upon his graduation from 
the agricultural program in the junior college? If the investi- 
gator is interested in knowing the type of position secured by 
the students who did not pursue further education, the question 
might be phrased: * 

“Did you begin full-time work when you graduated from high 
school?” Yes No 

“If you checked ‘yes’ in the foregoing question, please indicate 
what your first full-time job was.” 



(This second item could also be a multiple-choice one with the 
Jobs most commonly filled by high school graduates listed.) 
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The superiority of the open-ended question for seeking the data must be 
established. Although the appropriate situations when this type of question can 
be used are implicit in the introductory discussion, there is merit in further com- 
ment at this point. The key question that the investigator should ask himself 
in conjunction with this criterion ist “Could this question be converted ^ 
two-way or multiple-choice item and achieve the same result as this open-ended 
question?” A good general rule for the mail survey instrument is: If the question 
as a multiple-choice or two-way item is as good as the open-ended item, then one 
of the former types should be used. 

Example : “What type of course are you taking?” 

Comment: This question appeared on a survey instrument mailed. to former 
students of a junior college. Could not the investigator have 
listed ‘the most common “types of courses” students generally 
pursue in college and thereby have a better item? One further 

— - ambiguity is what the investigator means by “type of course.” 

Could he mean a two-year, a four-year, degree-granting course 
or a liberal arts, pre-medical, or history course? A multiple- 
choice item with the curricula offered in colleges would be a more 
appropriate item. The following is a suggestion: “In what cur- 
riculum are you now enrolled? 

Engineering 

Elementary Education 

Liberal Arts” 



etc. 

Example: “How are you occupied?” 

Comment: "From the nature of the survey instrument from which this item 
was taken, it appears that the investigator wanted to know the 
present occupation of the respondent. Answers such as: “going 
to school,” “working for A & B Company,” “doing secretarial 
work,” “in college,” etc., might very possibly appear in the ques- 
tionnaires returned. The investigator might have intended classi- 
fying present occupations according to some scale of job status. 
With such a variety of responses, this would be difficult to do. 
A multiple-choice item in which the level of response would be 
clear would be a far more effective item. A possible phrasing 
for the item is: 

“What is your present occupation? 

Clerical worker 

1- Gasoline station attendant 

Salesclerk” 

etc. 

Complex questions should be avoided. A question should ask for one item 
of information. As questions become more complex, the possibility of an inac- 
curate interpretation increases markedly. The investigator should not ask or 
expect the respondent to go through a series of mental calisthenics to arrive at 
the desired answers. 

Example: “How many periods are special counselors allotted in the school 

day to individual counseling and what is the period time in 
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minutes 2” 



Comment : The information desired should be asked for in two or more 
separate questions. The investigator should first find out if indi- 
vidual counseling is scheduled as a separate activity for the 
counselors. Then would come the question of how much time is 
spent in the activity, if it is provided. The length of the class 
period should be a still separate item. Three possible items to 
secure the information desired are: 

"Is individual counseling scheduled as a separate activity for 
your special counselors?” Yes No 

"If you answered the foregoing question by checking ‘yes/ please 
indicate the number of class periods allotted for the individual 
^counseling. .. 

- One period per day 

Two periods per day 

Three periods per day 

Four periods per day” 

etc. 

"How long is a class period in your school?” 

(in minutes) 

Example : "On the average (one month period) about how much time in 

hours and minutes does your child devote to homework each 
weekend period (Friday evening through Sunday evening) ?” 



Comment: Now, the parent who attempts to tackle this question must think: 
"How much time does Janie spend on her homework on Friday 
evening, Saturday (day and evening), and Sunday (day and 
evening) ? Does she spend about the same amount of time each 
week? Or, in one month, is she likely to be preparing for some 
sort of intermediate examination period?” Further ‘^bought of 
the parent attempting to answer the question could be imagined. 
Eventually, the keen parent comes to the question: "Am I to 
record the average amount per week, per day of the weekend, 
or the average for the month?” A far better question would 
have been one that asked the parent to record the amount of 
time Janie spent in studying last weekend. The ^investigator 
could determine the average per day, if he wanted it. The 
investigator’s question might well be: 

"How much time would you estimate your child devoted to 
homework during last weekend? (Friday evening through Sun- 
day evening)” (hours) 



The scope of the question must be indicated. The investigator cannot 
expect his respondent to go any further with his answer than the question 
implies. Questions do not necessarily stimulate the respondent to tell all. Many 
questions can be answered by the reluctant one with a “yes” or “no.” If the 
investigator wishes to know more, he should ask for more. 
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Example ; “Did you have a hard time getting a job?” 



Comment : If the investigator felt that this question would uncover the 
difficulties beginning workers have when seeking employment, 
he is going to be disappointed. A great number will answer 
“yes” or “no,” and the “why” will remain a mystery. Two ques- 
tions that would secure more complete answers from the respond- 
ents are: 

“Did you have any difficulty in securing a job when you gradu- 
ated .from high school? Yes No 

“If you checked ‘yes’ in the preceding question, please discuss 
briefly why you believe you had some difficulty in securing a 
job.” 



Example: “Can you swim, ride horseback, play ping pong?” 



Comment : A “yes” to this question as well as a “no” will tell the investigator 
litt’ i if anything about the recreational skills of his teen-age 
respondents. An “or” or “and” before “play ping pong” might be 
somewhat more enlightening. This item should be three separate 
ones or a multiple-choice item in which respondents are asked to 
check the skills they possess. The multiple-choice item might be: 

“Check the activities you are able to do. 

Swim 

Ride horseback 

Play ping pong” 

etc. 



MULTIPLE-CHOICE ITEMS 

By far the most popular items found on survey instruments are of the 
multiple-choice types. Such items require a minimum amount of writing; 
usually a check or a line is sufficient to indicate the response. When the choices 
possible are known, and there is a desire to get comparable data, the multiple- 
choice item is indeed appropriate. 

The two-way, dichotomous question is a commonly used multiple-choice type. 
The two-way question can be used when only two alternatives are to be con- 
sidered by the respondent. The “yes” “no,” ‘,‘app.rove” “disapprove,” “belief” 
“disbelief,” combinations are some of the alternatives used in such questions. 
Although the tv/o-way question appears exceedingly simple, closer inspection will 
often reveal that there are indeed other alternatives in addition to the ones listed 
that might be more appropriate for the respondent. 

Payne, in his discussion of the usefulness of the multiple-choice question, 
made the following comment: 

Multiple-choice questions are useful in two situations. The first of these 
is the case where the issue clearly splits into more than two parts, as blonde, 
brunette, or redhead. The second is the case where gradations are. asked. 
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for, 09 in very tall, tall, average, short, or very short. It may sometimes he 
difficult to distinguish between these two situations, hut the distinction is 
not too important. Where either variety or degree is under consideration, 
the multiple-choice, or ^‘cafeteria,*’ question has possible application.^ 

The variety of possible responses must be included in multiple-choice 
questions. Although there are occasions v/hen the investigator desires to “force 
a choice” on the part of the respondent, in general the investigator wishes to 
enumerate all the answers that the respondents would expect to find under the 
question. In cases where there is a possibility of a few other answers, it is satis- 
factory to add an “other” alternative, which permits the respondent to fill in an 
answer. The number of fill-ins should be very few if the item was well 
constructed. 

Example ; “Do you believe it is desirable to divide fast and slow learners 
* ^into-separate-classes-?— — — — — - 

Stro^ng yes 

Yes 

No 

Strong no 

No answer” 

Comment : Although the respondent .is given several gradations of possible 
feelings from which to choose, the question of the adequacy of 
“no answer” might well be raised. Parents responding to such 
an item would be inclined in some cases to want to say, “I don’t 
know.” The “I don't know” response would convey a very differ- 
ent attitude from the “no answer” alternative. 

Example : “Check those types of elections in which you consistently vote. 

Community government 

State government 

National government” 

Comment: Since elections for school board members and local school tax 
elections are often separate elections, should |;hey not be listed 
as separate items? 



The number of responses allowed the respondent must be clearly indicated. 
The investigator may want some indication of intensity of belief held by the 
respondent. In such an instance he, may want the item that the respondent 
would list first. However, unless the question indicates that one alternative only 
may be chosen, two or more may be selected. Many such returns would not be 
useful within the context of the investigator’s wishes. 

On the other hand, if the investigator wants to see how many are of 
importance or apply, then he should indicate the fact that a multiple answer is 
expected. 

Example : “Which of the following awards seem to you to be the most 

desirable to award to high school students? 



1 Stanley Payne, m (Piinceton: Princeton University 

^ess, 19Rx), p, 76. [In his discussion of the types of questions, Payne considers the 
^o-way question as a separate category. However, “multiple” means more than one: 
therrfore, the two-way question should be considered under the generic classification 
' multiple^noice* j 
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■ Athletic letters 

Service medals 

California Scholarship Federation pin 

Gold Seal 

Cardinal Key 

Scholarships 

Other - " 

Comment: Did the investigator want as many as the graduate felt were 
“most desirable”? If such were the case, it is very likely that 
the answers would range from one check to all items being 
checked. To secure some feeling of “most desirable” there might 
have been merit in asking for the “two most desirable” or the 
“three most desirable.” Only the most thoughtful respondents 
V70irld'Tittemp1rto~diff6rentiate m their own mihd^etween “most 
desirable” and just "desirable.” The others might have a tend- 
ency to check most of the items. 

Example : “Please check from the following list of attitudes or character- 

istics the ones you consider the most important. 

Cooperation 

Enthusiasm and willingness to learn 

Willingness to give a full conscientious day's work 

Promptness” 

(etc. — the list includes 19 different items) 

♦ 

Comment : The comment here is similar to the comment made about the 
preceding illustration.^ Here we have again a listing of items 
that many might consider “most important.” If the investigator 
wanted to know which one item came first in the minds of 
respondents, he would not get his answer here! 

The alternatives provided should be mutually exclusive. It is very con- 
fusing to the respondent to wonder where he should make his check when one is 
requested to check one item when two items are appropriate. 

Example : “What is your approximate income per month? (Please check) 

$100— $300 

$300— $500 

$500— $700 

i $700— $900 

^ $900— $1100” 

Comment : Where does the graduate who is earning $300 or $500, etc., check? 

If he checks the first alternative when he earns $300, he will 
ultimately understate the overall average. His difficulty does not 
match that of the investigator who attempts to determine where 
those with $200 income per month actually did indicate their 
salaries I 



Example : “Where do you get most of your information about your school 

system? 
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school 

newspapers 

friends and neighbors 

children” 

Comment : Where does the kindly taxpayer with no children but with many 
friends among the neighborhood children indicate her answer? 
There is a dual_ problem here, the investigator possibly meant 
by “children” “your children.” This should have been clearly 
stated ; one should not assume that such a meaning was intended. 

The number of choices should be limited to no more than six or seven items. 
Payne ^ speaks of the experiments that indicate the wisdom of limiting choices 
-to-no-more-than-sixr-There-arer'of-course 7 -instances-when~the~nature'“of "the 
question asked, makes it possible for the respondent to eliminate immediately a 
certain number of alternatives so that when contemplation begins, the respondent 
is in effect thinking of only 3 or 4 or 6 alternatives. If the question asked requires 
an evaluation, the inability of the respondent to differentiate between too finely 
identified gradations must impose a limitation on the investigator’s alternatives. 

Example : “Check those weaknesses you usually find in beginning workers: 

( ) lack of ability to follow orders 

( ) lack of ability to spell correctly 

( ) lack of ability to speak correct English 

( ) lack of ability to write correct English 

( ) lack of - ability ta do business arithmetic 

( ) lack of ability to meet people 

( ) lack of ability to get along with fellow workers 

( ) lack of ability to assume responsibility 

C ) lack of ability to be punctual 

( ) lack of initiative 

( ) unsuitable personal appearance” 

etc. 

Comment : To request an employer to consider such a variety of weaknesses 
is imposing a considerable burden. Indeed a far better idea 
would be to compose three or four questions in »which like items 
are grouped together. Such a plan would provide the respondent 
the opportunity to consider the weaknesses from a cluster of 
similar factors. Unwittingly the respondent, in reading the 
items as given, must evaluate the relative value of the factors. 
This introduces an aspect that the investigator may not have 
considered if his intention was “What are the weaknesses. . . .” 

Example : “My ability to use reference material has been: 

immeasurably increased 

considerably increased 

slightly increased 

not changed 

slightly decreased 

considerably decreased” 



*Ibid.f pp. 92-93. 
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Comment: This item to be checked by students who have been enrolled in 
an extended study workshop, for example, has more gradations 
than students may be able to differentiate. A three- or four- 
point evaluation might have been sufficient. There should, of 
course, have been a place to check “unable to determine. 



II. GENERAL CONSIDERATIONS IN WRITING ITEMS 

LANGUAGE ^ 

The wordS" used for conveying the intent of a particular Question must be 
carefully chosen. The general vocabulary level and the familiarity with the topic 
under discussion possessed by the sample to whom the instrument is to be sent 
must be kept in mind as words are selected. 



Example: “To what extent did the workshop experience develop creativity 

in your child? 

to a great extent 

to a limited extent” 

etc. 



Comment: The extent to which parents would understand what new be- 
havior of their children would represent “creativeness” is highly 
questionable. Even the professional educators are not in agree- 
ment on the manifestation of this attribute. A far more concrete 
term should be used in eliciting from parents the value of the 
experience for their children. 

Example: “Sociologists group families into various social classes. In what 

class would you rank your family? 

upper middle working lower” 



Comment: How realistic is it to expect high school youngsters to under- 
stand the class divisions represented by the alternatives ? Sociol- 
ogists have not yet agreed on one scale for classifying families. 
It would have been better if the investigator had asked young- 
sters to estimate family income, education of father, or some 
‘ other aspect of “social class standing” that would provide a clue 
to the investigator. 

Example: “Rank the following occupations in the order of prestige that 

' you attach to them. Give No. 1 to that occupation that you 
would rank the highest. No. 2 to the next highest, etc. 



_ laborer 

_ physical education teacher 
_ elementary school teacher 
_ truck driver 
_ physician” 

etc. 



Comment: “Prestige” is a word that has special meaning. To what extent 
~~ would high school students understand the sociological meaning 
of the word? An explanatory phrase following “. . . rank the 
highest,” would help to make concrete the basis of evaluation. 

Example: “Do you expect to improve your standard of living when you 
^ earn your own money? Yes No” 
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Comment : It is a conjecture to say what percentage of the ninth graders 
who were asked the above question would check “No.” How- 
ever, there is considerable question about their understanding 
of the meaning of “standard of living.” Many adults are hesitant 
about defining this economic term. 



GRAMMAR 

The principles of good grammar should not be violated in the writing of 
items for the survey instrument. An instrument will not receive the attention 
of the knowledgeable respondent if it is replete with grammatical errors;. The 
most common errors noted in survey instruments include: 



a. Lack of parallel construction in alternatives 



Example : “What are your students’ needs for counseling? Check 

Need for training to take responsibilities 

Failure to work to capacity 

^ Developing respect for authority 

Problems of physical health 

Financial problems 

Understanding of self and formation of future goals 

Getting along with people 

Others (list) . 



Comment : Although one may infer the “need” in each of the alterna- 
tives stated, the mixture of statements does not allow for 
smooth reading. Furthermore, from the question it appears 
that the alternatives will be items related to “Why the stu- 
dents come for counseling” and not “What they expect to 
receive.” Both types of items are included here. 

b. Failure to use complete statements 

Example: “Have home room teachers specialized training in guidance? 

yes or no 

Before assignment 

Are in-training courses provided 

Others (list)” 

Comment: Although the off-hand manner of asking questions represented 
in the foregoing example may be acceptable in informal 
speech, it is indeed not appropriate for a survey instrument. 
From the nature of the item, it appears that the investigator 
desires to know at what point homeroom teachers who have 
guidance functions in the school receive specialized training. 
The question should be complete. 



c. Lack of consistency in person used for questions 

Example: What is the married status of the student? 

( ) He was single when he enrolled and he is still single. 

( ) He was single when he enrolled but since enrolling he 
has married. 

( ) He was married when he enrolled. 
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Do you expect to return to college to continue your same 
education goal in the near future? 

( ) Yes 
( ) No 

Comment: In the survey instrument from which these two questions 
were taken, some items were directed to the individual an- 
swering the questionnaire, as in the case of the second 
tion above; others were written in the impersonal third 
person, as in the first question. 

PHYSICAL ARRANGEMENT OF ITEMS 

After care and attention have been given to the development of items, 
consideration must be allowed for making the items attractive to the respondent. 
In addition, attention to physical arrangement can expedite the task of trans- 
ferring the information when the instruments have been returned. Some sug- 
gestions for the investigator to remember as he devises his i nstr um ent are the 

following : 

1. Sufficient space should be allocated for answers that the respondent is 
expected 'to write in. Allowing one or two lines and then making the 
comment that the reverse side may be used for further discussion is not 
a good idea, unless the. question comes at the end of the form. 

2. Items should be lined up so that answers appear in as straight a column 

* as possible. At the point of transferring the data, the ease with which 
answers can be read from one vertical column as contrasted with reading 
answers from all parts of the page makes for much faster and more 
nearly accurate recording. 

3. A variety of type styles for emphasis and for ease of reading items 
should be used. A key word printed in italics can often provide a better 
idea of the meaning intended than a long descriptive sentence. A variety 
of type styles will also aid the reader in going from one section of the 
instrument to another. 

4 Adequate spacing should be provided between items to encourage the 

* reader to continue the task. The concern with keeping the questionnaire 
to a minimum number of pages should not preclude the consideration of 
the uncluttered page. Three pages attractively spaced is more likely to 
capture the attention of the respondent than is a two-page instrument that 
is so crowded that little unused space has been allowed. The finished 
instrument should be neat and orderly, and it should give the feeling 
that it is relatively easy to fill in. 



III. SUMMARY 

A carefully devised series of items' will assure the success of a survey. 
Some key considerations discussed in this chapter are given below. 

In writing open-ended questions, the investigator must 

a. determine the meaning of the question to the respondent. 

b. assess the superiority of the open-ended question for seeking the data 
desired. 

c. check the simplicity of the question. 

d. indicate the scope of the question. 
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In writing multiple-choice items, the investigator must 

a. provide the variety of possible responses to each item. 

b. indicate the number of responses expected. 

c. check the mutual exclusiveness of alternatives. 

d. limit the number of choices to no more than six or seven. 

Language, grammar, and physical arrangement of items are aspects of 
instrument development that also deserve attention by the investigator. 
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CHAPTER IV— ASSESSING THE ADEQUACY OF SURVEY INSTRUMENTS 

Pages 

INTRODUCTION ' 60 

I, THE MEANING OF RELIABILITY AND VALIDITY 60 

Reliability 
Validity 

Some general considerations 

II. evaluation of the survey instrument in accord- 
ance WITH CRITERIA GOVERNING ITS CONSTRUCTION - 62 

III. VALIDATION CRITERION AND CRITERION TEST ITEMS 52 

Criterion — The survey results agree with some criterion which is 
regarded as an acceptable measure of the phenomena 
being studied. 

0 12 3 4 

Criterion Test Items 

Rating 

1. Do the survey results agree with subsequent events ? 

2. How do the survey results compare with results 

using some other research technique? 

3. Has a pretest and an analysis been made of the ques- 
tions in the survey instrument? 

4. How do the survey results compare with objective 

records (if it is feasible to make such a comparison) ? 

6. Has a comparison been made of responses to an 

existing situation with those to a nonexisting one? 

6. Have expressed attitudes or statements been checked 

against specific behavior or actual practice? 

7. Have the survey results been compared with the 
judgments of experts or persons qualified to evaluate 

the situation? — 

8. Have tests been made of groups known to be biased, 
who were included in the survey population, to 
determine whether or not the survey results reveal 

such biases? 

9. Has the significance of no-response or fio-opihion 

responses to the survey questions been investi- 
gated? — 

10. Has the validity of the survey data met the test of 

"common sense?” ' 
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IV. RELIABILITY CRITERION AND' CRITERION TEST ITEMS 65 

Criterion — ^A -repetition of the survey using the same data-collection 
■ procedures and covering a comparable sample population 
gives the same or appr oxima tely the same results. 

0 12 3 4 



Criterion Test Items 

1. Has the reliability of the survey data been tested by 
a repetition of the survey with a selected sample of 
the original survey population, or a comparable 
group, after a short interval of time? 

2. Does the survey data gathered by two equivalent or 
parallel forms of the survey instrument compare 
favorably? 

S. How do the responses given to two forms of the 
same question in the survey instrument compare? 

■ 4. ' Does an odd-even comparison of question results 
give a favorable correlation? 



Rating 



V. SUMMARY OF SPECIAL CHECKS TO BE MADE OF ALL SUR- . 

- VEY INSTRUMENTS 67 

Criterion — ^A variety of special checks are made of the survey instru- / 
ment, of the survey data, and of the survey population S 
in assessing the adequacy of the survey instrument and ; 
the collected data. 

0 12 3 4 



Criterion Test Items 

f. Has ah analysis been made of the preliminary form 
of the survey instrument after tryout with a repre- 
sentative pilot group? 

2. Has the survey instrument been evaluated by recog- . 
nized research authorities or by an advisory com- 
mittee? 

3. Has a final careful revision and inspection, of the 
survey form been made? . 

4. Have special validity and reliability checks been 
made of the survey data? 



Rating 



6. Has a preliminary “editing” been made of the com- 
pleted survey instrument forms before the data are 
tabulated? 

6. Has an analysis been made of the respondent pop- 
ulation to determine if a proportional representation 
of the total survey population is included in the 
study? 

7. Has an analysis of the non-respondent group or sub- 
groujw been made as an indication of certain cau- 
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tions that may need to be observed in the interpre- 
tation of the collected data? 

8. Have plans been made for efficient tabulation of 
* the collected data and the making of any needed 
statistical analyses of such data? 



VI. SUMMARY 
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CHAPTER IV 



ASSESSING THE ADEQUACY OF 
SURVEY INSTRUMENTS 

Jshoda ^ has indicated that the function of any data-collection technique 
“is simply to produce precise and reliable evidence which is relevant to the 
research questions being asked,” The earlier chapters of this report stress that 
it is important that the researcher choose the best means for collecting needed 
or necessary data. Again, it may need to be emphasized that a survey instru- 
ment is but one of the tools of research that provides for the collection of data 
frbm which a variety of decisions, interpretations, or actions can be derived. 

I. THE MEANING OF RELIABILITY AND VALIDITY 

Reliability. Reliability is usually referred to as the accuracy and the con- 
sistency of the measuring instrument in securing data from^time. to time. A 
reliable survey instrument should, therefore, yield comparable data upon 
repeated administration to the same population ; it is not unlike a yardstick or 
other calibrated measuring device which gives the same measurement, if used 
properly, each time it is applied. A special measuring device would tell us, for 
example, that the Empire State Building is 1,250 feet in height not including 
the 222 additional feet of television antennae attached to the top. This measure- 
ment would only vary slightly, if at all, when repeated again and again. Such 
high level reliability is impossible to attain, however, with any survey instru- 
ment. 



Validity. A survey instrument would, in the strict sense, be considered valid 
if the data gathered were a true measure of what the instrument was designed 
to measure. A swimmer frequently will dip his foot into the water to test the 
temperature of the' water before he jumps or dives in. To the extent that his 
foot IS sensitive to the temperature of the water, it may be considered a valid 
measure or indicator of the overall temperature of the water. The sensations 
which the swimmer’s brain received from his foot as he dipped it into the water 
would represent the rough data relative to the temperature of the water. Upon 
this evidence, he may make his decision as to whether or not the water is too 
cold for swimming. 



Some g;eneral considerations. Herzog 2 has stated that “Reliability is pre- 
requisite to- validity. . . . Yet no amount of reliability in itself establishes valid- 
ity. Perhaps in relation to, the validity of a survey instrument, the question 
needs to be asked, “Validity 'for what?” If a survey instrument were used to 
assess skill deficiencies of beginning office workers, for example, and if a par- 
ticular category were concerned with spelling ability, we would need to define 
quite carefully just what is meant by spelling ability before we could judge the 
validity of the questions making up this category. Is spelling ability, for instance, 
the ability to single out incorrectly spelled words as in proofreading? Or is it 
the ability to recall the correct spelling of words needed in the transcription of 
shorthand notes or machine dictation? Next, we would need to ask, “Why is this 



1 Mane Jahoda, Mort^ Deutsch, and Stuart W. Cook, Research Methods in Social 
ReUUom (Part I: Basic Processes; New York: The Dryden Press, 1961), p 9^ 

.. TT Herzog, Some Guide Lines for Evaluative Research (U. S. Denartment 

of Health, Education, and Welfare, Social Security Administration, 1969), p. 41. 
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information of importance or how will we use it to modify or irpprove educa- 
tional practice?” We need answers, therefore, which will provide direct or 
positive evidence of the kind of skill deficiency rather than an overall skill defi- 
ciency if the greatest value is to be gained from the analysis of the data. In a 
similar vein, if clerical aptitude were being assessed, we would need to ask, 
“Aptitude for what? For general office work? For .statistical typewriting? For 
meeting the public? For accuracy in recording? etc.” ^ 

A school district may want to assess the success of its evening program for, 
adult office workers in terms of improved job performance. Herzog, in treating 
this problem from' another viewpoint, discusses sonie important considerations 
which indicate that it is often possible to amass reliable evidence, but very diffi- 
cut to demonstrate ite validity: 

The validity, of findings about job performance will depend (a) on the 
extent to., which -job performance actually reflects improvement and (b) on 
~<ihe extent to which the indicators selected and the way in which they are 
appUM actually reflect job performance. 

To start with (b), there is room for debate on the validity of indicators 
. .used to- reflect job improvement. Neither a quantitative nor a qualitative 
: rheasure is likely to be enough in itself: Increased quantitative production 
at the sacrifice of quality may not represent improvement. Improved quality 
.at the exorbitant sacrifice of quantity may not represent improvement. 
These problems can be handled, but they suggest that to work out valid 
indicators of job improvement is by no means simple. 

. . .. under certain circumstances better performance on a routine 
manual job could conceivably result from an individual’s giving up aspira- 
tions to a more satisfying career for which he is in fact well suited. . . even 
though improved job performance is reliable beyond question, its validity as 
a major criterion of improvement would remain in doubt.^ 

The reader may well ask at this point, as Herzog suggests, “What, then, is the 
ultimate evidence on which validity rests? How can one prove that^ a person is 
better off (or worse off .or unchanged) after, let us say, an educational experi- 
ence?" (Italicized words those of the authors of this report.) Herzog answers 
this question as follows: 

For the time being, at least, it appears thnt “objective" proof is not 
available. The most “operational" or “behavioral" definition of outcome is 
based on someone’s conviction about what is desirable or undesirable, what 
is adjustment or maladjustment, what is improvement or deterioration. The 
■ “ultimate" criterion of success or improvement is an opinion. . . . Validity 
requires, then, that all measures and categories be explicit and theoretically 
tenable, and that the use of all measures and categories be reliable.* * 

The validity dilemma may, in part, be solved by exactness in defining and 
wording the categories and questions to be used, coupled with a consistency in 
applying them, to make sure that everyone who is to receive the survey 
instrument will understand what is wanted. Herzog ® again cautions that, “Lack 
of validity at any point can disturb the validity of the whole; and we have 
not yet achieved ability to test, control, and demonstrate validity at every point.” 
It is necessary to recognize that absolute reliability or validity cannot be attained 



^ Ibid., p. 42. 

* Ibid., p. 48. 
^ Ibid., p. 49. 
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even with so-called objective facts. Factual data, of course, ha.ve a higher degree 
of vaiidity and reliability, but such data may 

securing evidence which may help in the solution .of the particular problem with 
which the survey is concerned. Limited findings can be very useful, P^ov^ed i^he 
limitations are recognized=-and clearly stated and provided, also, they are consid 

. ered in the interpretation of the findings. and 

From all this emerges the need to recognize the specificity of validity and 

reliability. A survey instrument may be valid and reliable only ^ 

carefully defined population; its validity and reliability may be affected by sex 
differences, interest differences, and the like. What may be- concluded then is 
that the most important: general test of the validity and reliability of a sur^y 
instrument, in any given instance, is, local validation by the survey user in term., 
pf his own particular purposes for making the survey. 

IT evaluation of the survey instrument in accordance 

WITH CRITERIA GOVERNING ITS CONSTRUCTION 

The researcher should make certain that the survey instrument has been 
developed in accordance with the criteria and criteria test items governing is 
construction. These are listed in Chapters I, II, and III of this report According 
?o Ryans,« the design and wording of the survey instrument s a matter of prime 
importance and one upon which the validity and reliability of the data will 
depend. The governing criteria and the criteria test items, designed to help 
insure such validity and reliability,, are given in considerable detail in the 

Prev j^®^(j(jition, the researcher may subject the rough draft of the survey 
instrument to Kornhauser and Sheatsley’s » check list of points to consider in 
formulating questions and to Payne’s » “test of the meaning of words. 

HI. VALIDATION CRITERION AND CRITERION TEST ITEMS® 

n’TTT? QTTPVPV "RESULTS AGREE WITH SOME CRITERION WHICH IS REGARDED 
CRITERION: THE^SURVBY^ RBSULT^^ Ef THE PHENOMENA BEING STUDIED. 

To determine the validity of a survey instrument, the research worker is 
always concerned with the degree to which the survey results agree with some 
acceptable or evaluative criterion. It is not the intent of this report to discuss 
the statistical procedures by which the obtained results may be correlated with 
the ^^true” situation in determining the validity of the findings. For all practical 
purposes, when survey instruments are used in securing needed data in connec- 
tion with some local educational problem such detailed analysis is not necessary. 



CRITERION TEST ITEMS 

Some of the common tests which may be used to test the validity of survey 
instruments are given here. 

1 The survey results may be compared with subsequent events. The 
comparison of survey results with subsequent events may be especially important 
when an opinion-attitude survey instrument is used. Often there are differences 



« David G. Ryans (Unpublished Syllabus in Research Design, University of Cali- 

fornia,^Los Angeks^l9^ B. Sheatsley, “Gmde 

tion,” Appendix C, as included in Claire Selltiz, et s\.,R^earch 
tions (New York: Henry Holt and Company, Inc., 19^), pp. 652-574. 

8 Stanley Payne, The AH of Asking Questions (Princeton: Princeton University 

Press^, 1961^,ig.^l41.pai^en, Surveys, Polls, and Samples: Practical Procedures, pp. 
486-493, especially. 
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between what respondents say they “think they will do” and what -they “really 
will do.” The same' may be true when a comparison is made between what 
respondents “say they do” and “what they actually do.” Jt is possible to make 
additional checks to determine such discrepancies or differences. For example, 
the influence of a consumer education course on the buying habits of persons 
who haye had the course could be tested at a later date by a check of the 
purchases actually made by the respondents. The agreement between what a 
Pierson says he “intends to buy” and what he “actually does buy” need not be 
100 percent to be an adequate test of the validity of a survey instrument 
designed to determine buying habits, since considerations such as having the 
money to buy or to make down payments may enter into the final purchase of 
consumer goods regardless of the consumer’s desire in the matter. 

2. The survey results may be compared, with results obtained using some 
other research technique. It is always wise, if possible, to check results obtained 
by survey instruments with results obtained by a selected number of interviews 
with respondents who may be considered as a representative sample of the 
population being surveyed. 

3. The questions used in the survey instrument may be pretested and 
analyzed. A pretest of the questions designed for use in a survey instrument 
often will reveal such things as lack of clarity of definitions, ambiguity or varying 
meanings of some words to different groups of people, words which have an 
emotional overtone, questions which may be misleading or misinterpreted, the 
need for no-opinion or no-response categories, of the omission of pertinent 
questions that may be of value in interpreting responses made to other 
questions. 

4. The survey results may be checked against objective records. When 
a fact-finding survey instrument is used, it is often possible to check a sample 
of the returns with the actual records, or with the objective facts as revealed 
by means of interviews, selected observations, and the like. 

5. A comparison may be made of responses to an existing situation with 
those to a nonexisting one. The inclusion of false items in a survey instrument 
may also be used to give an indication of the validity of the returns. If a survey 
were being made to determine the reading habits of office workers, for example, 
some of the categories may include sonie false book titles. The number of false 
items checked by the respondents would give an indication of the validity (truth) 
of the responses. It is relatively easy to provide for and to make such validity 
checks in every survey study. 

i 

6. Expressed attitudes or statements may be checked against specific be- 
havior as revealed by follow-up interviews or other tests. Expressed attitudes 
are often in conflict with specific behavior or actual practice. For example, 
representatives of business enterprises frequently indicate that they prefer the 
liberal arts graduate of collegiate institutions to the graduates who may have 
studied business administration or some other specialty; however, their expressed 
attitudes are not -always in accord with their specific behavior. A study of 
employment policies usually indicates that they still hire more college graduates 
in terms of a specific specialty than they do liberal arts graduates. Survey 
findings which indicated, therefore, that the business employers preferred liberal 
arts graduates to other kinds of graduates would give misleading information 
which could not be considered valid evidence of employment practices. Here, 
again, is another kind of validity check that it is important to make if meaningful 
and valid data are to be presented in the survey research report. 
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Another example where stated practice may not be in accord with actual 
practice is the following'. Business employers frequently indicate that they want 
stenographers and secretaries who can take dictation at 120 words a minute, 
but when questioned they have little knowledge of .a dictation speed of 120 words 
a minute. A tape recording and analysis of the dictation speeds used in regular 
office work usually shows that dictation is given at much slower spee'ds than 
120 words a minute. 

7. Survey results may be validated by comparison with the judgments of 
experts or persons qualified to evaluate the situation. Success is often evalu- 
ated in terms of job status. Many companies' have found that they have been 
able to reduce the turn-over problem in some positions-toerely by changing the 
title for that -position. The true job situation in some*^cases, then, may not be 
reflected by the job title. In other instances, respondents have been known to 
give misleading information about job positions during interviews. The classic 
example is that of a census taker who was interviewing selected members of the 
population with reference to varying aspects of their economic, social, and 
educational backgrounds. One housewife told the census taker that her husband 
was a bank director. The interviewer became suspicious because the home situa- 
tion did not reflect the level of living that may have been expected of the typical 
bank director: Further investigation did reveal that the husband was indeed a 
kind of “bank director” — ;he directed customers of the bank to the various tellers, 
banking departments, and executives. 

These examples indicate the need for and the importance of special checks 
of survey results in areas where the survey results, although not inaccurate, 
may not be entirely valid in terms of revealing the true or existing situation. 

8. Tests may be made of groups known to be bjased to see whether or 
not the survey results reveal such biases. Again, when an opinion-attitude sur- 
vey instrument is used, and if bias is suspected, it is wise to check to see if the 
survey results accurately reflect any such biases. One method for doing this 
has been suggested by Lundberg.io It involves securing a consensus from selected 
persons included in the survey as to which of their close friends, who were also 
included in the survey population, hold pronounced attitudes on certain questions 
in the survey form. A comparison of the responses of such persons to certain 
questions on the survey form with the judgment of close associates as to their 
true attitudes t^ill give the researcher an indication of the degree of validity of 
the responses made by these same individuals on the survey form. 

9. The significance of no-response or no-opinion responses to survey ques- 
tions should be investigated. Especially when an opinion-attitude survey instru- 
ment is used, a high “no-opinion” response to an item should lead the investigator 
to suspect the validity of the particular question. The question may be phrased 
in language which is too technical for the average respondent. Where this hap- 
pens, the question doesn’t measure what it was designed to measure. The ques- 
tion may actually be a more nearly valid measure of the educational background 

’ of the respondent. 

Another possibility is that the question or item may not be phrased “nat- 
urally” or in the manner in which people ordinarily respond to the situation. 
In other words, the investigator may not understand how people actually talk 
about the topic being investigated. This may be the reason, therefore, for the 
nonresponse. 



George A. Lundberg, Social Research (rev. ed.) (New York: 
and Co., 1942), as quoted by Mildred Parten, op. cit., p. 492. 



Longmans, Green 
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10. Common sense may be used as a measure of the validity of survey 
data. Eecently a-secretary in a business office was asked to get the height in 
feet of certain office buildings in the New York City area. Her report of the 
height of the Empire State Building cimmediately 'revealed a common sense 
inconsistency. It was reported as 1,250 feet in height, with an overall height 
of 4,720 feet. The latter height included the television antennae that are 
attached to the top. Common sense would immediately indicate that there was 
an inconsistency in the figures since it would be rather difficult to attach more 
than a half-mile of television antennae to the top of a building, despite the 
remarkable engineering feats that we have come to accept as commonplace in 
recent years. A recheck of the data indicated that the overall height was in error. 
Common sense estimates, or judgments may, however, be misleading; therefore, 
such estimates or judgments cannot be used as the sole test of the validity of a 
survey instrument. 

The foregoing include many of the tests that may be made of validity. 
For many survey studies, it would neither be necessary nor desirable., to make 
all these tests'of the instrument to insure a usable degree of validity. 

IV. RELIABILITY CRITERION AND CRITERION TEST ITEMS 

CRITERION: A REDEMPTION OF THE SURVEY USING THE SAME DATA-COLLECTION PRO- 
CEDURES AND COVERING A COMPARABLE SAMPLE POPULATION GIVES THE 
SAME OR APPROXIMATELY THE SAME RESULTS. 

A survey instrument yields reliable data to the extent that the findings 
reflect true data that are free from variable error. A survey instrument is 
reliable if the respondents answer the same questions in the same way when 
the questions are asked at different times or are varied in their phrasing, 
assuming the phrasing is equally clear in both instances. 

The preceding chapters have suggested various criteria which are important 
in the selection and design of survey instruments. If the instrument conforms 
to these criteria it will, at the outset, have a considerable degree of internal 
reliability; then, depending upon the care with which it is completed by the 
survey..population, it may also have a high degree of external reliability. Relia- 
bility of a specific survey instrument may be expected to vary somewhat if 
readministered to the same or a comparable population. The differences that 
may be expected will depend upon such factors as the4ype of survey, the method 
of collecting the data, the kinds of respondents (sex differences, for example, 
often influence reliability — male responses are often different from female 
responses and the combined responses may give yet another degree of reliability), 
the interval between surveys, the interest and motivation of the respondents to 
give truthful or accurate answers, the care with which the form is completed 
by the respondent (when a considered opinion is desired it may be necessary to 
request a delayed response rather than an immediate response to the survey 
items), and/or the strengths of various conditions that may be operating 
between the initial survey and its reapplication. 

To summarize, some of the factors which may influ ence th e reliability of 
survey research data are the fallowing: 

1. The number of questions asked. (Generally speaking and within limits 
which the reader will recognize, the greater the number of questions 
pertaining to a particular category, the more reliable the resultant data 
will be.) 

2. The time spent and the care exercised by the respondent in completing 
the survey form. 



“ See Mildred Parten, op. cit, pages 494-498, especially. 
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3. The objectivity of the plan for recording the responses of the survey 
population. 

4. The homogeneity of the objects or responses measured. _ 

5 The number of'\conditions which may lead to misinterpretation. 

' tional coloring of words used in directions or in survey items, inadequate 
or faulty directions, misleading intent of questions or directions— all 

6. Th?\ncento motivation of the respondent to furnish accurate 

information. , . ,, 4. o 

7. The effect of previous mental or emotional experiences on the part 01 a 

respondent to the survey item. 

8. The desire of the respondent to give truthful and accurate responses 

(to avoid cheating or faking). ■, 4. . 4.- 

9. The stability of the situation in which behavior is observed from time 

10. The^Txtent to which a representative sample of the population can be 
induced to complete the survey form and return it. 

CRITERION TEST ITEMS . 

The more commonly used methods for testing the reliability of survey instru- 
ments are listed and’ discussed briefly in this section. 

1. A repetition of the survey may be made with a selected sample of the 
original survey population, or a comparable group, after a short interval of 
timl The reliability of the survey data may be determined (1) by readmin- 
istering the survey form with a selected -sample of the survey Population after a 
period of time and by making a comparison of the data with the o^^^?J^oal , 
or, (2) by comparing the data collected in the pretest of the survey JJ^trument 
with the data subsequently collected. The degree of correlation of the results 
will be an indicator of the reliability of the survey instrument 

The preceding test assumes, of course, that no systematic changes have 
taken place between the time the survey form was first administered and its 
readministration. Such an assumption is not always tenable. 

2. A comparison may be made of the survey data gathered by two equiva- 
lent or parallel forms of the survey instrument. Two equivalent or parallel 
forms of the survey instrument may be administered one after the other to the 
same population, or to two similar samples of a survey population simultaneously. 
The greater the correspondence between the survey results, the more confidence 
the analyst may place in the findings of the survey instrument. 

3. A comparison may be made of- the responses given to two forms of the 
same question included in the survey instrument. If the two forms of the same 
question are equally clear, a comparison of the answers given by the respondent 
to such questions would give an indication of the reliability of the survey item. 

4 An odd-even comparison of question results may be made. If a series 
of questions relating to a single issue or category can be summarized to form 
a single score such as an attitude score toward some current issue, the scores 
obtained when only the even-numbered items are employed can then be corre- 
lated with the scores resulting from using the odd-numbered items only. Here, 
usually, a special statistical analysis such as the use of the ^Spearman-Brown 
formula is required to determine the reliability coefficient existing between the 
two parts. It can be seen at once, however, that this test has limited application 
in terms of use with most survey instruments. Furthermore, the split-half 
method often leads to spuriously high reliability coefficients. 
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As is true for determining the degree of validity of a survey instrument, 
the degree of reliability may be computed statistically for the reliability tests 
suggested in the foregoing discussion. These specialized statistical techniques 
are discussed in most textbooks on statistical research procedures. For many 
research studies in which a survey instrument is used, it is often not necessary 
•to make such a complete analysis of the reliability of the instruments as a 
data-collection device. 



V. SUMMARY OF SPECIAL CHECKS THAT NEED TO BE MADE OF ALL 
SURVEY INSTRUMENTS 



CRITERION: A VARIETY OF SPECIAL CHECKS ARE MADE OF THE SURVEY INSTRUMENT, 
OF THE SURVEY DATA, AND OF THE SURVEY POPULATION IN ASSESSING THE 
ADEQUACY OF THE SURVEY INSTRUMENT AND THE COLLECTED DATA. 



A number of special checks should be made in assessing the adequacy of a 
survey instrument and the data collected by means of the survey instrument. 
These special checks are one indication that the investigator is aware of the need 
to take such precautions as will increase the validity and reliability of the 
collected data as well as the confidence with which the research findings may 



be viewed. 



CRITERION TEST ITEMS 

The following special checks should be made of the survey instrument, of 
the survey data, and of the survey population in assessing the adequacy of the 
survey instrument and the collected data. 

1. Pilot group tryout and analysis of preliminary form of the survey 
instrument. A test or tryout of the preliminary survey instrument with a repre- 
sentative pilot group is usually considered necessary and advisable before the 
survey instrument is put in final form. Such a test can help the investigator 
determine the practicability of the instrument in terms of such questions as 
these: (1) Will the intended survey population respond to such a questionnaire 
or survey instrument? (2) Can their responses be interpreted as an indication 
of the existing situation? (3) What questions may need rephrasing or change 
on the basis of the respondent’s reaction to the question? (4) What additional 
questions or categories may need to be provided in order to secure all necessary 
data relating to the survey problem? (5) What defects are there in the instruc- 
tions used or in the form of the survey instrument? (6) What is the overall 
reaction of the pilot group to the survey instrument? 

The pilot group may be asked not only to complete the survey form but 
also to comment, criticize, or make suggestions about it. If the latter procedure 
is followed, then these reactions should be tabulated and analyzed in preparation 
for necessary revisions which may need to be made in the survey instrument. 

In addition, the data collected by means of this preliminary survey form 
need to be studied and analyzed. Rough tables should be drawn up to determine 
if the recorded responses can be tabulated satisfactorily and if the data of the 
tables will provide significant answers to the major questions with which the 
investigation is concerned. 

2. Expert judgment of the survey instrument by recognized research 
authorities or by an advisory committee. Recognized research authorities may 
be asked to review the survey instrument and make comments or suggestions for 
its improvement; however, here it must be recognized that such authorities 
rarely have the time to do an extensive amount of such advisory or consultative 
work. Often, the State Department of Education may offer the services of its 
various research specialists in connection with survey design and development, 
or these specialists may be asked to react to the survey instrument and to offer 
comments or criticisms. 
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In his own school district, the researcher may organize an advisory com- 
mittee and hold discussions with this group relative to the survey instrument 
as it is being designed and developed. Such committees frequently can offer 
much valuable assistance. 

The researcher, however, must recognize that others also have busy schedules 
of work; it is not proper to ask them to do work which it is his job to do. 

3. A final careful revision and inspection of the survey form. After the 
survey instrument has been evaluated through use with a pilot group and by 
research authorities or an advisory committee, a number of revisions may have 
to be made. The final inspection of the survey instrument should be in terms of 
such things as (1) content; (2) form; (3) sequence and number of questions; 
(4) spacing, arrangement, and appearance of the items; (5) adequacy of direc- 
tions for completing the instrument; (6) need for coding of items to simplify 
the tabulation of the returns; and (7) proper identification of the survey 
instrument in terms of such items as the title of the study, date, origin or source 
of the instrument (name of author or authors, school district or other educa- 
tional institution), and other pertinent or useful information which may be 
used for classification purposes at some later date. After these checks have been 
made, the final survey instrument and the covering letter may be prepared. 

a. Physical appearance of survey form . The format, the style of printing 
to be used, the use of color, and the general physical appearance of the final 
survey instrument are all of importance. The final survey instrument should 
look as professional as the budget will allow. This is an important consideration 
from the point of view of inducing the survey population to respond to the form. 
In addition, as has been suggested earlier, the items should be arranged so that 
it is relatively easy for the respondent to complete the form. The use of color, 
provided the cost is not prohibitive, often gives an added attractiveness to the 
physical appearance of the form which may increase the percentage of returns, 
an important consideration in any survey research study. 

b. The covering letter or letter of transmittal . The covering letter sent to 
the survey population with the survey instrument should explain the nature of 
the study and should attempt to arouse an interest in the study. Often the 
basic appeal used in covering letters is “for information as a personal favor.” 
Such letters, as Lundberg has suggested, are usually more productive of 
results than letters offering a trivial reward or stressing some alleged advantage 
to the respondent for answering. If the letters can be written on the official 
letterhead stationery of a school district, the percentage of returns may be 
increased. The covering letter usually should not be over one page in length. 
If the data requested is of a confidential or personal nature, complete anonymity 
of the respondent must be assured. A typewritten letter, because of its personal 
appeal, is usually more effective in getting returns than is a duplicated letter. 
If, because of number, the covering letters are. duplicated, a handwritten post- 
script adds a personal appeal that is effective in increasing the percentage of 
returns. Finally, a fixed date for the return of the completed survey instrument 
should be suggested in the closing paragraph of the letter. 

4. Special validity and reliability checks. In the interest of time, cost, and 
energy, the researcher may find that the following special validity and reliability 
checks of the adequacy of the survey instrument are sufficient: 

a. Consistency of the survey responses as revealed by a comparison of the 
responses on the preliminary and the final survey forms. In the interest of an 



12 George A. Lundberg, Social Research (rev. ed.) (New York: Longmans. Green 
and Co., 1942), p. 186. 
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accurate interpretation of the data collected by means of a survey instrument, 
it is important that some analysis be made to determine the consistency of 
response” as it relates to the data provided by the preliminary survey form and 
the final survey form. Although the final survey form may differ from the 
preliminary form, the comparison can be made on the basis of those questions 
which remained the same or essentially the same on both forms. The number 
of consistent and inconsistent responses to the same or similar questions can be 
tabulated and a “consistency-of-response” percentage can be. computed. This 
percentage will not indicate complete agreement, but it will indicate whether or 
not it may be necessary to make additional checks to determine reasons for 
the inconsistencies as these may affect the interpretation of the data. Such 
analysis will always add to the confidence with which the survey results may 

be viewed. . ^ • 

b. Consistency of survey responses as revealed by selected interviews with 

a representative sample of the survey population . It is generally good research 
practice to check the conformance of data gathered by means^ of a survey instru- 
ment with data as may be determined by intensive and detailed interviews with 
a representative sample of the survey population. Such interviews may provide 
clues such as the following: (1) the consistency or inconsistency of responses 
and first-hand information about the causal factors leading to such consistency 
or inconsistency; (2) the respondent's attitude toward the survey instrument and 
the type of information it was designed to provide; and, (3) whether or not the 
respondent attempted to give or did give accurate responses to all questions 
included in the survey instrument. 

5. Editing of survey instrument returns. Before the data on the completed 
survey forms are tabulated, the following steps are often necessary as a pre- 
liminary editing of the survey instruments: 

a. If a respondent classification sheet is used with the survey form, this 
sheet should be inspected to see if the special classification data needed 
for purposes of the study have been furnished. 

b. The survey instruments should be checked to determine whether or not 
different par.ts of the survey responses are consistent. 

c. The responses to questions should be quickly reviewed for any apparent 
errors on the part of the respondent. 

d. The responses and/or comments on each question should be inspected 
to bring the used, or the intended, summarizing categories into line 
with the comments written on the survey form or with the additional 
answer categories which the respondent may have used. 

e. If the survey instrument qualifies for inclusion in the study after each 
of the foregoing steps, it may need to be numbered or coded for control 
purposes. 

The survey instruments which may otherwise have qualified for inclusion 
in the tabulation, but which were judged as incomplete according to the fore- 
going suggested checks usually are not used in the study. They should be classi- 
fied as non-returns or unusable returns in computing the “percentage of re- 
turns” in the study. 

6. Analysis of the respondent population according to the returned survey 
instruments which are usable in the study. The researcher will rarely get a 
100 per cent return of survey forms which are sent through the mails. Every 
effort should be made through follow-up letters and cards to secure as high a 
percentage of return as possible as this adds to the confidence that may be 
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given to the interpretations of the collected data. Eventually, however, if the 
study is to be completed within a reasonable time limit, the resarcher will have 
to reach a cut-off point on the returns and complete the tabulation of the data 
collected. The assumption is made, of course, that the percentage of returas at 
the cut-off point is sufficiently high and that a proportional representation of 
the total survey population is included in the returned survey forms to give the 
degree of confidence that is necessary if meaningful interpretations are to be 
made of the collected data. 

7. Analysis of non-returns in each of the subgroups of the survey pop- 
ulation. If the survey population has been classified into various subgroups, 
an analysis of the non-respondent population may indicate cautions that need 
to be exercised in interpretating the data collected. If significant subgroup 
classifications of the survey population failed to respond to the survey instru- 
ment, some effort should be made to determine the reason for such non- 
response. This analysis of the non-returns may be important to the meaningful 
interpretation of the cbllected data. 

On the basis of the preceding two special checks, the researcher can make 
important decisions as to whether or not the respondents who actually returned 
usable survey forms are typical of the total survey population. 

8. Tabulation of the collected data and making any needed statistical 
analyses of such data. The researcher is always charged with the responsibility 
of presenting evidence of the appropriateness of the techniques used to tabulate, 
compare, and analyze collected data. The readers of his final report will then 
have an indication of the degree of confidence that may be given to any of -the 
research findings that are presented in the report of the study. 

a. Tabulation of data with special reference to coding. Usually in advance 
of sending out the final survey instrument, the researcher should have reached 
some decision as to the method that is to be used in the tabulation of the 
collected data. Rarely is it convenient to deal- with data in the form in which 
they originally have been obtained. 

Hand tabulation of data, when the survey population is large, is a laborious 
and tedious process. It is subject to recording or tabulating error. Such data- 
processing errors (errors in coding, tabulating, calculating, or consolidating 
data) may seriously affect the reliability and validity of the collected data. 

Developing a system for coding questions on the survey instrument and 
coding the collected data is an aid in tabulation of the data, especially if 
machine tabulation is used. Coding is the assignment of numbers or symbols 
of some sort to the answers to questions or any kind of data that falls into 
some specified class. Coding simplifies the sorting and reassembling of the data 
into specified classifications according to some pre-determined characteristic of 
the data which may be useful in making interpretations of the data. 

The coding system to be used must be pre-determined in view of (1) the 
nature of the data and (2) the kinds of cards, and sorting, to be employed. 
The coding system may be a simple, direct system, or may, in the interest of 
efficiency, be quite ingenious, involving combinations of code characters to serve 
various purposes in compiling the data of the study. Ryans offers these sug- 
gestions for coding: 

(1) The code should he carefully prearranged, so there is no possibility of 
the assignment of data to a class as a matter of judgment. 

(2) If the data are to be hand sorted, various symbols and colors may be 
used for coding as well as numbers. (For punched card sorting and 
tabulation, all data must be coded in numbers.) 
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(S) In dealing with numerical data, the data should be reduced to a simple 
common unit. 

(U) The rounding off procedure to be used in dealing with numbers must 
be prearranged when the code is developed. 

(5) Provisions should be made with a code number or symbol to indicate 
lack of response to a question or absence of data regarding a particular 
characteristic. 

(6) It is usually better to use a more detailed' classification at the time the 
data are being recorded than the classification that may be used later 
in summarizing the research data. (Data from smaller classifications 
can always be combined, but a broader classification cannot be broken 
down into its component parts once the data are recorded.) 

(7) Coded d)ata should be verified by an independent check.^^ 

An illustration of a simple coding system that may be employed with 
responses to multiple-choice items in a survey instrument is given below. 

9. Do you plan to go to college? Please check one of the responses given 

below. 

1 ( ) I have definitely decided NOT TO GO to college. 

2 ( ) I am UNDECIDED whether or not to go to college. 

3 ( ) I plan to go to college ULTIMATELY but WILL NOT GO 

NEXT YEAR. 

4 ( ) I have DECIDED TO GO to college and WILL GO NEXT YEAR. 

5 ( ) Other (Please specify). 



The preceding illustration shows the multiple-choice responses that are listed 
for a specific category that might be found in a survey instrument^ "concerned 
with the educational plans of high school seniors. The numbers given just prior 
to the check-list response items represent the simple code that would be useful 
in the tabulation of the responses. In the tabulation of the responses, these 
code numbers would be used with this series of questions in the survey instru- 
ment. These code numbers have no qualitative values, however. 

Various types of cards may be utilized for the recording of collected data 
so that they can be conveniently tabulated and used later in a variety of ways 
when interpretations of the data are to be made. Some of these cards are the 
following; 

(1) Uncoded data cards. Uncoded data cards may be mimeographed or 
printed with spaces designed for recording various kinds of informa- 
tion and data. 

(2) Marked-edge cards. Marked-edge cards are cards, approximately 5 by 8 
inches in size, on which spaces around the four edges can be assigned 
for recording various characteristics of the data. The data on such 
cards may be recorded in numerical form or by qualitative description, 
such as sex, class in school, occupation, etc. In addition, they may be 
coded for convenient sorting by marking the edges of the card either 
in black or with various colors. When this is done, the cards can readily 
be hand sorted into various classifications. 

(8) Punchedredge or key-sort cards. Punched-edge or key-sort cards are 
similar to marked-edge cards except that they are especially prepare^ 



Ryans, op. cit., p. 169, 
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(manufactured) with a row of holes around each edge. Each group of 
holes may be assigned to a particular characteristic, such as sex, test 
score, response to a specific question, etc. Then a special punch may be 
used to make U-shaped notches indicative of certain identifying infor- 
mation that the researcher plans to use. The cards may then be sorted 
by running an ice pick or a similar instrument, or a special long needle, 
through the deck of cards. Those cards punched to reflect a given char- 
acteristic fall out of the deck. This is a quick and efficient classification 
system which gives the researcher ready access to data m making a 
variety of analyses. 

(4) Punched-hole cards. Punched-hole cards are the rather well-known 
cards such as those used by IBM and other data-processing machines. 
The use of such cards requires pre-planning and coding of the survey 
instrument data in advance. They also have certain limitations which 
the researcher would need to discuss with a data-processing repre- 
sentative, if he were planning to tabulate the data by a machine 
process. 

(5) Punched or special magnetic tape. Punched or special magnetic tape 
represents the newest method of recording or tabulating data. Such 
tape is now used with the newer electronic data-processing machines. 

The tedious task of hand tabulating the data collected by means of a survey 
Instrument can be eliminated to a large extent by adapting the data to a system 
of elecronic data processing. The facilities of data processing centers at various 
colleges and universities are available to certain qualified educational institutions 
and research workers. Before starting any study which may require the use of 
such facilities, it would be wise to first consult with a research specialist at one 
of the' nearest colleges or universities, or the researcher may secure information 
by writing directly to the nearest data processing center. One such center 
is located at the University of California, Los Angeles; another is located at 
the Massachusetts Institute of Technology, Boston. Usually there is no charge 
for machine time or any other special data processing service, except for occa- 
sional c.o.d. mailings to a user of large numbers of tabulating cards or magnetic 
tapes. Also, for a fee, special arrangements may be made with various data 
processing machine manufacturers for the tabulation of the data of a study. 
These arrangements should always be made in advance of the mailing of the 
survey instruments to prospective respondents so that proper programming and 
coding schedules can be prepared and the survey instruments coded correctly 
for ease in machine tabulation of the collected data. 

b. Statistical treatment of data . Unless the researcher has an extensive back- 
ground in statistics, it may be well for him to consult with a research statistician. 
Such a specialist could indicate any statistical analyses that may need to be made 
if the data are to be properly interpreted and evaluated in terms of statistical 
significance. (Basic statistical reference books which may be of value to the 
researcher are listed in the Selected References at the end of this report.) 

Best expresses a note of caution concerning the use of statistics: 

Statistics is an important tool of the research worker, and an under- 
standing of statistical methodology and terminology is important for the 
consumer of research. There are a number 'of limitations, however, that 
should be remembered in using statistical processes, in interpreting statistical 
processes, and in drawing conclusions from statistical evidence. 

1. Statistical process is the servant of logic and only has value if it 
verifies, clarifies, and measures relationships that have been established by 
clear, logical analysis. Statistics is a means, never an end, of research. 
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2. A statistical 'process should not be emplo'yed in the analysis of data 
unless the basic assumptions and limitations underlying its use are clearly 
understood. 

3, The conclusions derived from statistical analysis 'will be no more 
accurate or valid than the original data. All of the refinement of elaborate 
statistical manipulation 'will not yield significant truths if the data result 
from crude or inexact measurement. 

U. All treatment of data must be checked and double-checked frequently 
to minimize the likelihood of errors in measurement, recording , tabulation, 
and analysis. 

5. There is a constant margin of error 'wherever measurement by human 
beings is involved. This error is increased tohen qualities or characteristics 
of human personality are subjected to measurement, or 'when inferences 
about the population are made from measurements of statistical samples. 

When comparisons or contrasts are made, a mere number diffefence is, 
in itself, not a valid basis for any conclusion. A test of statistical significance 
should be employed to 'weigh the possibility that chance in sample selection 
could have yielded the apparent difference. To apply these measures of 
statistical significance is to remove some off the doubt from the conclusions.. 

6, Statisticians and liars are often equated in humorous quips. There 
is little doubt that statistical processes can be-used~to prove nearly anything 
that one sets out to prove. Starting 'with false assumptions, using inap- 
propriate procedures, or omitting relevant data, the biased investigator can 
arrive at false conclusions. These conclusions are often particularly danger- 
ous becouse of the authenticity that the statistical treatment seems to 
confer.'^-* 

A knowledge of basic statistics, however, is a valuable tool for any research 
worker and is desirable for anyone engaged in educational work. Especially 
important to the researcher, too, is a knowledge of statistical sampling tech- 
niques that may be effectively utilized in any survey study. The researcher 
planning an important survey study involving sampling should either study the 
pertinent literature or consult a statistical specialist, preferably both. 

Finally, it is necessary to express a note of caution, too, when percentages 
and the differences in percentages are made the basis for comparisons of data. 
When the total number of cases to be compared is less than 100, it is important 
to recognize that the addition or subtraction of a single case in any one 
category may unduly affect the resultant percentage figure. It is important to 
recognize this- fact if correct interpretations are to be drawn from tables which 
give percentage figures as a basis of comparison. 

VI. SUMMARY 

When using a survey instrument of any type, the researcher will always 
need to know if the instrument will yield the kind of data required for the 
study (validity), and the extent to which the collected data are free of 
extraneous factors so that there is a consistency of measurement (reliability). 
To determine the validity of a survey instrument, the research worker must 
compare the survey results with some criterion of known validity which is 
related to the phenomenon being studied. Similarly, by a variety of checks, the 
reliability of the instrument must be determined. 

Before a survey instrument is prepared in its final form, it should be given 
a variety of special checks, such as (1) tryout with a pilot group which is 

John W. Best, Research in Education (Englewood Cliffs, New Jersey: 

HaU, Inc,, 1969), pp. 243-244. 
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considered to be representative of the survey population selected for the study; 
(2) evaluation by research specialists and/or an advisory committee; and (3) 
special validity and reliability tests. 

The completed returns need editing in terms of the usability of the data 
furnished; an analysis needs to be made to determine the adequacy of the 
respondent population in terms of the total survey population which has been 
included in the study; and, finally, the data collected need to be classified and 
tabulated in a convenient form for their interpretation and analysis. When 
this has been done, the research worker is ready to draw conclusions relative 
to the findings and to write the research report. „ — 
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CHAPTER V— REPORTING THE RESULTS OF SURVEY RESEARCH 

Pagea 

INTRODUCTION 

I. GENERAL REPORT WRITING CONSIDERATIONS 68 



Criterion— The research report is written in a clear, logical, and 



readable style. 
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Criterion Test Items 

Rating 

1. Does the report conform to fundamental principles of 

English composition? . 

2 . Does the report have an interest appeal and is it 

written with the reader in mind? c 

3. Does the report follow an acceptable form or outline? 

4. Has time been allowed in the writing schedule for 
necessary revisions that must be made before the final 

copy of the report is prepared? 



II. GENERAL PUBLICATION CONSIDERATIONS 

Criterion — ^The research report is published in some form, or 
plans are made for publication, within a reasonable 
period of time after the completion of the survey 
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Criterion Test Items 

Rating 

o - — 

1. Have plans been made for report -publication and 

has a “deadline” date been established? 

2. Have plans been made relative to the form in which 

the report is to be duplicated? 



HI. CLASSIFICATION, ANALYSIS, AND INTERPRETATION 
OF SURVEY DATA AS A PRELIMINARY STEP TO WRIT- 
ING THE RESEARCH REPORT 



Criterion— The collected data are properly classified and analyzed 
so that valid and reliable interpretations can be made 
of the data. 
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Criterion Test Items 


1. Have the collected data been adequately classified as 

an initial step to interpretation? 

2. Have the data been thoroughly analyzed as a second 

step to interpretation of the data? 
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Pages 

Subtests ; 

a. Have tables, charts, graphs, and other summariz- 
ing forms been used for graphic presentation of 

the data? 

b. Have the findings been analyzed in terms of their 

possible significance and importance? 

c. Have immediate notes been made of implications 

that occur as the data are studied? 

d. Has there been an awareness of the possible psy- 
chological effect of the survey instrument upon 

the respondents? 

IV. PREPARATION OF CONCLUSIONS 75 

Criterion — The conclusions are a logical outgrowth of the 
analysis and interpretation of the data 

0 12 3 4 

Criterion Test Items 

Rating 

1. Do the conclusions answer the questions which gave 
rise to the survey and are the conclusions limited to 

the data and the interpretations of the data? 

2. Has there been strict adherence to the rules of logic 

and straight thinking in arriving at the conclusions? 

3. Are the conclusions consistent with other known 

facts? 

4. Have all facets of the data been considered in arriv- 
ing at the conclusions? 

5. Have the conclusions been developed with a concise- 
ness and a clarity that is apparent to the reader? 

6. If any of the conclusions represent generalizations 
from the interpretation of the data, has this been 

made clear to the reader? 

7. Have the tentative conclusions been subjected to 
various tests as a measure of their soundness before 

the final draft of the conclusions is prepared? 



V. ORGANIZATION OF THE RESEARCH REPORT ON PAPER 78 

Criterion — Principles of correct presentation (style) and re- 
search report organization (form) are followed in 
writing the research report. 

0 12 3 4 

Criterion Test Items 

Rating 

1. Has a consistent style been followed in all parts of the 

report? 

2. Has an acceptable format or research outline been 

followed ? 






o 

ERIC 



£ 

j 

/>« 



Evaluative Ceiteria 



VI. FINAL EVALUATION OF THE SURVEY RESEARCH 
’ REPORT 

Criterion— The research report is subjected to a final critical 
analysis and evaluation before publication or before 
it is duplicated in its final form. 0 12 3 4 

Criterion Test Item _ : . 

Rating 

Has the research report been subjected to an overall 

final critical analysis and evaluation? 

VII. SUMMARY 



Pages 
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CHAPTER V 



REPORTING THE RESULTS OF 
SURVEY RESEARCH 

This chapter presents the formal steps in the preparation and writing of a 
survey research report. Modifications in this formalized approach may need 
to be made in a survey report designed to provide data on a specific educational 
problem of a localized nature. The steps presented and the suggestions made 
are intended to provide guide lines and criteria of value to the survey research 
worker. 

I. GENERAL REPORT WRITING CONSIDERATIONS 

Making the necessary and proper interpretations of research data and writ- 
ing the research report are matters that are much like the problem of “all the 
King’s men” when they tried to put Humpty-Dumpty together. There is no 
magical word, such as abracadabra, that will do the trick. The process is chal- 
lenging and interesting to the research-oriented person not so much because of 
the certainty of success, if the necessary time and effort is expended, but because 
of the promise of discovery. 

CRITERION: THE RESEARCH REPORT IS WRITTEN IN A CLEAR, LOGICAL, AND READABLE 
STYLE. 

It has often been said that, “If research is worth doing at all, it is worth 
reporting so that it can and will be read.” Someone has said, too, that the 
research reporter “stops writing English when he begins reporting research.” 

CRITERION TEST ITEMS 

Some of the report writing considerations which may be used as guide lines 
are included in the discussion which follows. 

1. Does the report conform to the fundamental principles of English com- 
position: Does it have unity, coherence, and emphasis? One of the tests that 
might be made of a research report is to check that the various parts conform 
to the 'basic principles of English composition. Unity, coherence, and emphasis 
should be apparent in the sentences and paragraphs making up each chapter. 
One chapter should logically follow another. 

A clarity of style which includes unity, coherence, and emphasis, requires 
that the reporter possesses a good basic knowledge of English grammar. It is 
easy to find examples of “tangled sentences,” incoherent paragraphs, and illogical 
organization in research reports. 

If the writer does not have a sense of structure, he can hardly expect the 
reader to understand his pattern of thought. Often, too, the research reporter 
forgets that his long acquaintance with the problem under consideration enables 
him to see relationships that may not be apparent to the reader. He, therefore, 
must make sure that the structure of the whole report is clear to the reader. 
The best solution is to use clear sentences that hang together and present an 
uncluttered picture that gradually unfolds as the reader makes his way through 
the report. Thus it is that the reader travels the straight road, with no detours. 

The research reporter who has difficulty expressing himself clearly will find 
many valuable suggestions in E. B. White’s ^ interesting chapter titled “An Ap- 
proach to Style.” 

1 William Strunk, Jr. and E. B. White, The Elements of Style (New York: The 
Macmillan Company, 1969), Chapter V, pp. 62-71. 
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2. Do 68 the report have an ‘interest appeal and is it written with the 
reader in mind? It is important to keep the reader in mind at all times when 
writing the research report. What sort of person will he be? How much back- 
ground information will he have about the particular problem which is discussed 
in the report? How can the data be presented so that it will have an interest 
appeal? 

What might be “dull research data” can come alive in the hands of the skill- 
ful reporter. The foregoing statement does not imply that the research reporter 
must write in the racy style frequently employed by the fiction writer. There 
are several principles of good writing that may be employed to make the report 
interesting. Some of these principles are listed here: 

a. The vrincvple of 'personalization. The principle of personalization means 
putting human interest into the research report by the use of specific 
examples and illustrations as the data are presented. It may be well to 
remember that respondents to a survey instrument are people, even after 
they have been dehumanized by statistical treatment. They can be used 
to personalize the report. The review of related literature can be a story 
of other research success and failure. It can point up the need for the 
present study. There can be drama in methodology. The research reporter 
can tell how he collected and treated the data. He can use illustrative cases 
to make concrete the generalizations derived from statistical treatment. 
He can explain and describe deviations from central tendencies. 

b. The principle of design and structure. The survey instrument must have 
a clear design and structure. The research report, too, must meet this 
standard. This design and structure should be made clear to the reader. 
For example, out of his experience and study, the research reporter has 
formulated certain hsrpotheses regarding the problem which is the subject 
of the report. These hypotheses should be stated. The research reporter 
can then describe his plan for testing the hypotheses, or how he has 
treated the data collected. The presentation of the findings of the study 
then naturally follow with their interpretation and the generalizations or 
conclusions that are warranted. 

c. The principle of emphasis. The reporter should make certain that the 
reader gets the important points. These can be “highlighted” by means 
of headings and subheadings. Too frequently the main ideas are cleverly 
disguised behind a facade of words which are scattered on the page in a 
scrambled jigsaw style. When this is done, the reader has the almost 
impossible task of “rattling the research brush pile with the hope that he 
can scare out a few rabbits which may represent the findings of the 
research.” 

The reader, too, should be led along a straight-forward path to compre- 
hension. Although the reporter should never make unwarranted conclu- 
sions from the data being presented, research reports are too often 
cluttered with “it would seem,” “it should be recognized,” “it was demon- 
strated,” “one might judge,” and other somewhat meaningless expres- 
sions. Translation of this cumbersome kind of writing into a terse, active 
form would help to vitalize many research reports. 

Another factor related to the principle of emphasis is to present and space 
ideas so that the reader has time to grasp them. The space between main 
ideas can be used for specific illustrations and supporting details which 
build up to main ideas. 

d. The principle of plain words. Choice of words is important in making a 
report readable. First, the reporter should avoid “empty words” that do 
not express essential ideas; emotional words which may color or detract 
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from an objective presentation of the data; and words which may have 
dual meanings. Second, as suggested by Rudolf Flesch in his book The 
Art of Plain Talk, the reporter should use short, simple, concrete words 
whenever possible. If difficult technical words are necessary, the reporter 
should try to use them in a context that makes their meaning clear, or 
he should define and illustrate such words when they are used. 

3. Does the report follow an acceptable form or outline? The logical 
presentation of the research data and findings is facilitated if an acceptable 
research outline is followed. Suggested outlines are found in many books on the 
preparation, writing, and planning of research reports (see the Selected Refer- 
ences at the end of this report). Section V of this report illustrates one such 
acceptable outline and the divisions which are generally used. 

In addition, there are various mechanical aids to clear report presentation. 
One such aid is the footnote. Footnotes can be used for a variety of purposes, 
such as (1) acknowledging quotations from the work of others, (2) giving credit 
to others who have been concerned with the same problem, (3) telling the reader 
where he can learn more about the particular issue being discussed, and (4) 
e^laining points which may be somewhat obscure in the report itself. Another 
aid is to divide the report into chapters and sections and to use appropriate 
headings, subheadings, side headings, and paragraph headings. A third aid is the 
use of tables, charts, graphs, maps, and other graphic devices for classifying and 
summarizing data. 



4. Has time been allowed in the writing schedule for the necessary revi- 
sions that must be made before the final copy of the report is prepared? It 
would be the unusual person who could prepare an acceptable copy of the research 
report on its first draft. In writing most research reports, it is often best to 
prepare an initial rough draft which can then be refined and checked to see 
that it conforms with the general writing considerations mentioned in this 
section. Time must be allowed for this necessary revision work in the writing 
schedule. The researcher may want to have others read the report in its various 
stages of development and pass on its acceptability. 

As each page of the copy is prepared, the pages should be numbered for 
proper identification of the order of the copy. This may be done in pencil if 
additional pages are to fie added later, or the additional pages may be given 
code numbers to identify the section where they are to be inserted when the 
final copy is prepared. If a number of drafts are necessary, it is suggested that 
each typed page bear a page number, chapter number, draft number, and date. 
Usually it is good practice to keep all draft copies (properly segregated, of 
course) of the report for some time after the final copy is prepared. These 
copies may have value in answering questions that may arise at a later date. 

Prior to^ the final typing of the report, the copy should be laid aside tem- 
porarily. This ‘‘cooling off” period may enable the researcher to discover gaps 
in the report, additional implications of the data, or other weaknesses when 
he re-edits the report for clarity of style and organization, sentence structure, 
paragraphing, and wording. At this time, too, unnecessary material can be 
pruned from the report. This final critical study and analysis of the report 
should be a must on the part of the research reporter. 

II. GENERAL PUBLICATION CONSIDERATIONS 



CRITERION; THE RESEARCH REPORT IS PUBLISHED IN SOME FORM, OR PLANS ABE WAnE 

FOR PUBLICATION. WITHIN A REASONABLE PERIOD OF TIM^^ 

COMPLETION OF THE SURVEY STUDY. ujr TlMi. AFTER THE 



Research is not complete until it has been reported and, if possible, pub- 
lished. Often survey research is undertaken in an attempt to seek answers to 
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3 ome local educational problem in a particular school district. In such cases, the 
report of the research is frequently written for local consumption only , none- 
theless, if such research has implications for educational practice in 
districts the reporter has a research obligation to prepare a summary or abstract 
wMcht Shouts to an appropriate educational journal for Possible P^. 

Ucation. Knowledge of such r^earch and the findings may save other school 
districts time, effort, and money if they are contemplating similar studi^, o i 
the SngTm be adapted to the solution of specific educational problems in 

their own school districts. 



CRITERION TEST ITEMS 

Some Of the considerations that are important in terms of report publica- 
tion are listed below. 

1. Plans should be made for report publication and a “deadline” date 
should be established. The value of most research lies in its 

fore, plans should be made for some form of publication of 
relatively short period after the survey ^ta have been collected. If it can be 
assumed that the survey study was of sufiicient importance to be 
is equally important that time, funds, and personnel be budgeted for the prepara- 
tion S the written report so that it can be published. When the survey is under- 
taken, the date of publication should be set and an appropriate time schedule 
should be developed to insure that this “deadline” date is met. 

2. Plans should be made relative to the form in which the report is to be 
duplicated. It is important that a decision be reached before the wntmg of t 
report begins as to the form in which the report is to be duplicated. Depending 
upon the number of copies which may be needed, the research report ^® 
typed mimeographed, multigraphed, printed, or duplicated in a variety of other 
wlirs ’ Each process has special problems with respect to such items as arrange- 
ment’; spacing; use of colors; cost of reproducing tables, charts, and graphs, and 

0 ? Wbn should be decided after, consideration of 
the relative costs and merits of the various processes. The choice of the method 
of duplication will depend upon such factors as (1) the number of copies desired 
and the possibility of the need for additional copies at some later d^e, (2) the 
number and types of tables, illustrations, charts, graphs, photographs, or other 
spedal de^^^^^^ to be included in the report; (3) the format desired, 

number of pages, and number of words to the page; (4) the price at which the 
report is J sold (if it is to be sold); (5) the type of reader to whom ^ the 
report is addressed; and, (6) the importance of a “finished-product look in 

relation to the purpose and importance of the report. ., j. j 

If the reseLch is sponsored, the sponsoring agency may provide funds for 
the publication of the findings in a special printed report, or a monograph, so 
lhat therwil^^^^^^ to a larger public; it may also rssume the response 

bility for the distribution of the completed report. . 

The purpose of the survey and the potential readers of the final repojt of 
the survey results must be considered, too. When the report is Prepared^ 
nonular cLsumption by readers not familiar with survey procedures or termi- 
noloJTV the plan emphasis, and style may have to be modified somewhat from 
?h^SoyTforTse with educational specialists. The usual practice is to 
uremre a special summary report which highlights the findings when the report 
L prepared^^for a group of readers not directly concerned with the problem of 

the survey study. 
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Again, it should be emphasized that if the study may be of interest and 
value to others in the educational field, every effort should be made to get at least 
an abstract of the report published in an appropriate educational journal. 

III. CLASSIFICATION, ANALYSIS, AND INTERPRETATION OF SURVEY 

DATA AS A PRELIMINARY STEP TO WRITING THE RESEARCH 

REPORT 

CRITERION: THE COLLECTED DATA ARE PROPERLY CLASSIFIED AND ANALYZED SO 
THAT VALID AND RELIABLE INTERPRETATIONS CAN BE MADE OF THE 
DATA. 

The classification, analysis, and interpretation of the survey data constitute 
important steps in the research process. It is this basic study of the data which 
is used to support the implications or conclusions that may be drawm from the 
study. If the survey research instrument has been properly designed, if the 
collected data have been adequately classified, analyzed, and presented in a logical 
and orderly manner, the conclusions drawn from the data will help to provide at 
least some of the answers to the questions that formed the basis for the study, 
or they will lead to the acceptance or rejection of the hypotheses which guided 
the research. 

It is at this point in the study that the research person needs to ask himself 
this question: How good are these findings? If the data are valid and reliable, 
the answer is provided not by the findings alone, but by the findings plus the 
interpretation that is made of the findings. It is at the point of interpretation 
of data that the research skill and insight of the research person is tested. He 
must be able to make valid and reliable interpretations of the tabulated and 
classified data, and he should see the implications of such interpretations. 

CRITERION TEST ITEMS 

What, then, are some of the preliminary steps that need to be taken if the 
findings are to be properly classified and analyzed so that valid and reliable 
interpretations can be made of the data? 

1. The collected data must be properly classified as an initial step to 
interpretation. Proper classification of data has a variety of functions which 
may be listed as follows: 

a. It is the first step in quantitative description or analysis of data. 

b. It provides for organization of collected data. 

c. It is a form of definition in that it makes possible a more precise descrip- 
tion of a thing, a concept, or a category. 

d. It suggests possible relationships between groups of data. 

e. It is basic to the quantitative or statistical analysis of data. 

Usually at the time the survey instrument is designed, the researcher will 
attempt to set up classifications which divide the collected data into relatively 
homogeneous groups or categories. Often, too, even though useful classifications 
have been built into the survey instrument, additional classifications of the data 
are suggested at the time the data are tabulated. If the research instrument has 
not been designed so that the collected data will be pre-classified, it will be 
necessary to establish suitable and usable classifications of the data at the time 
they are tabulated. 

Some of the principles governing classification of data may be listed as 
follows : 

a. Classification may be based on any common characteristic of the data to 
be sorted or classified. (The characteristic may be one selected for con- 
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venience, resulting in an artificicbl classification, such as, an alphabetic 
classification; or, it may be a ncbtuTObl characteristic of the data- to be 
classified, such as, kinds of work experience.)^ 

b. Classification requires clearly defined categories. 

c. The various classes must be mutually exclusive (must not overlap). 

d. The various classes must he comprehensively exhaustive. (A place must 
be provided for every possible datum, including no response.)^ 

e. Classification presupposes extensive knowledge of the characteristics and 
properties of the data being classified. 

f. Classification may be simple or complex. (For example, the number of 
students in each high school class— freshmen, sophomores, juniors, and 
seniors — could be counted. This would be a simple classification. A more 
complex classification would be a division of the class groups into male, 
female, and age groups.) ^ 

2. After the data have been properly classified, they should be thoroughly 
analyzed as a second step leading to accurate and meaningful interpretation 
of the data. After the data have been classified, the important question is this: 
How are the data analyzed? There is no one best way to answer this question. 
Some suggestions which may be of value to the research person are these: ^ 

a. Use tables, charts, graphs, and other summarizing forms for presentation 
of the data . Data are often best analyzed if the classifications of the data are 
presented in a series of tables, charts, graphs, or other summarizing forms. 
Such devices are also of value to the reader as they enable him to comprehend 
and interpret masses of data rapidly, and to see significant details and relation- 
ships. 

When tables are used, the survey data are often converted into percentages 
as a basis for comparison and analysis. Such a procedure, however, may require 
special attention to the interpretation. For example, as was noted earlier in 
this report, when the number of cases is less than 100, the change of one case has 
an important effect upon the percentage figure; in some complex classifications 
(when the classifications may not be mutually exclusive) the sum of the percents 
may total more than 100 percent. Data in tables may be subjected to a variety 
of statistical analyses to test the significance of any differences noted. 

The procedure usually followed by survey reporters is to examine the tables 
in the order regarded as most important at the time of the tabulation of the 
data. Parten suggests that for each table, the analyst asks such questions as 
these : 

(1) What is the most important point revealed by this table? 

(2) What other points are brought out? 

(S) What is significant about the total shown? 

(Jf.) Are the averages higher or lower than one would have expected or 
than were known to have been the case at an earlier period or in 
another place? 

(5) Are the averages pretty stable or is there great variability among 
them and in the components from which the average is computed? 

(6) Why do the smallest figures shown fall in that particular grouping? 
How can the infrequent or low numbers be explained? 

(7) What is the largest figure shown? Does it make sense? How does it 
compare with numbers or percentages found in comparable items, 
places, populations, on the same table or on other tables? 

2 Adapted from David G. Ryans (Unpublished Syllabus in Research Design, Uni- 
versity cf California, Los Angeles, 1964), p. 88. 



